Hanzo Sentry - Error Tracking & Performance Monitoring

Hanzo Sentry is a self-hosted Sentry 24.2.0 deployment for error tracking and performance monitoring across all Hanzo, Lux, Zoo, and Pars services. It is a fork of getsentry/sentry with a custom Do...

Overview

Hanzo Sentry is a self-hosted Sentry 24.2.0 deployment for error tracking and performance monitoring across all Hanzo, Lux, Zoo, and Pars services. It is a fork of getsentry/sentry with a custom Dockerfile branded for Hanzo, configured for OIDC SSO via hanzo.id, and deployed to the hanzo-k8s cluster.

Why Hanzo Sentry?

Self-hosted: Full control over data, no SaaS dependency

Multi-tenant: Single instance serving four domains (hanzo, lux, pars, zoo)

IAM integrated: OIDC SSO login via hanzo.id (app-sentry client)

Secrets via KMS: All credentials synced from kms.hanzo.ai

OTEL instrumented: Exports traces to the shared OTEL collector

Shared infrastructure: Uses hanzo-sql (PostgreSQL) and shared Redis

Tech Stack

Backend: Python 3.11 (Django 5.x, Celery, DRF)

Frontend: React + TypeScript (Webpack, Emotion CSS)

Queue: Celery via Redis (or RabbitMQ if configured)

Database: PostgreSQL via hanzo-sql.hanzo.svc

Cache: Redis via redis.hanzo.svc

Image: ghcr.io/hanzoai/sentry:latest

Base version: Sentry 24.2.0 (FSL-1.0-Apache-2.0 license)

OSS Base

Fork of getsentry/sentry. Local clone at ~/work/hanzo/sentry/. The GitHub repo hanzoai/sentry does not yet exist as a public fork -- currently origin points to getsentry/sentry and Hanzo customizations live in the self-hosted/ directory and K8s manifests in universe/infra/k8s/sentry/.

When to use

Instrumenting Hanzo services with error tracking

Monitoring performance of API endpoints, workers, and frontends

Viewing crash reports, stack traces, and breadcrumbs

Setting up alerting rules for error spikes

Tracking release health across deployments

Hard requirements

PostgreSQL via hanzo-sql.hanzo.svc (database sentry)

Redis via redis.hanzo.svc (cache, queues, rate limits, buffers)

KMS at kms.hanzo.ai for secret sync (secret-key, db-password, oidc-client-secret)

IAM at hanzo.id with app-sentry OIDC client for SSO

Quick reference

Item	Value
UI	`https://sentry.hanzo.ai`
Alt domains	`sentry.lux.network`, `sentry.pars.network`, `sentry.zoo.network`
Port	9000 (web), internal K8s service
Image	`ghcr.io/hanzoai/sentry:latest`
Base version	Sentry 24.2.0
Language	Python 3.11 + TypeScript
K8s namespace	`hanzo`
Deployments	`sentry-web` (2 replicas), `sentry-worker` (2 replicas), `sentry-cron` (1 replica)
Database	`hanzo-sql.hanzo.svc:5432/sentry`
Redis	`redis.hanzo.svc:6379`
Auth	OIDC SSO via hanzo.id (`app-sentry`)
KMS project	`hanzo-k8s-epiq`, path `/sentry`
Local clone	`~/work/hanzo/sentry/`
K8s manifests	`~/work/hanzo/universe/infra/k8s/sentry/`
Health check	`GET /_health/`

One-file quickstart

Instrument a Python service

import sentry_sdk

sentry_sdk.init(
    dsn="https://<key>@sentry.hanzo.ai/<project-id>",
    traces_sample_rate=0.1,
    environment="production",
    release="my-service@1.0.0",
)

Instrument a Node.js service

const Sentry = require("@sentry/node");

Sentry.init({
  dsn: "https://<key>@sentry.hanzo.ai/<project-id>",
  tracesSampleRate: 0.1,
  environment: "production",
  release: "my-service@1.0.0",
});

Instrument a Go service

import "github.com/getsentry/sentry-go"

func main() {
    sentry.Init(sentry.ClientOptions{
        Dsn:              "https://<key>@sentry.hanzo.ai/<project-id>",
        TracesSampleRate: 0.1,
        Environment:      "production",
        Release:          "my-service@1.0.0",
    })
    defer sentry.Flush(2 * time.Second)
}

Docker Compose (local development)

# compose.yml
services:
  sentry-web:
    image: ghcr.io/hanzoai/sentry:latest
    command: ["sentry", "run", "web"]
    ports:
      - "9000:9000"
    environment:
      SENTRY_SECRET_KEY: "${SENTRY_SECRET_KEY}"
      SENTRY_POSTGRES_HOST: postgres
      SENTRY_DB_NAME: sentry
      SENTRY_DB_USER: sentry
      SENTRY_DB_PASSWORD: "${SENTRY_DB_PASSWORD}"
      SENTRY_REDIS_HOST: redis

  sentry-worker:
    image: ghcr.io/hanzoai/sentry:latest
    command: ["sentry", "run", "worker"]
    environment:
      SENTRY_SECRET_KEY: "${SENTRY_SECRET_KEY}"
      SENTRY_POSTGRES_HOST: postgres
      SENTRY_DB_NAME: sentry
      SENTRY_DB_USER: sentry
      SENTRY_DB_PASSWORD: "${SENTRY_DB_PASSWORD}"
      SENTRY_REDIS_HOST: redis

  sentry-cron:
    image: ghcr.io/hanzoai/sentry:latest
    command: ["sentry", "run", "cron"]
    environment:
      SENTRY_SECRET_KEY: "${SENTRY_SECRET_KEY}"
      SENTRY_POSTGRES_HOST: postgres
      SENTRY_DB_NAME: sentry
      SENTRY_DB_USER: sentry
      SENTRY_DB_PASSWORD: "${SENTRY_DB_PASSWORD}"
      SENTRY_REDIS_HOST: redis

  postgres:
    image: postgres:16
    environment:
      POSTGRES_DB: sentry
      POSTGRES_USER: sentry
      POSTGRES_PASSWORD: "${SENTRY_DB_PASSWORD}"

  redis:
    image: redis:7-alpine

Core Concepts

Architecture

                  ┌──────────────────────────────────┐
                  │         K8s Ingress               │
                  │  sentry.hanzo.ai                  │
                  │  sentry.lux.network               │
                  │  sentry.pars.network              │
                  │  sentry.zoo.network               │
                  └───────────────┬──────────────────┘
                                  │
                  ┌───────────────▼──────────────────┐
                  │   sentry-web (2 replicas)        │
                  │   Django + uWSGI on port 9000    │
                  └──┬────────────┬─────────────┬────┘
                     │            │             │
              ┌──────▼──┐  ┌─────▼─────┐  ┌───▼────────┐
              │ Postgres │  │   Redis   │  │ OTEL       │
              │ hanzo-sql│  │  (shared) │  │ Collector  │
              └─────────┘  └───────────┘  └────────────┘
                     │            │
              ┌──────▼──┐  ┌─────▼──────────────┐
              │ sentry- │  │ sentry-worker       │
              │ cron    │  │ (2 replicas, Celery) │
              └─────────┘  └─────────────────────┘

Three Deployments

Component	Command	Replicas	Purpose
`sentry-web`	`sentry run web`	2	Django web server (uWSGI)
`sentry-worker`	`sentry run worker`	2	Celery async task processing
`sentry-cron`	`sentry run cron`	1	Scheduled tasks (digests, cleanup)

Multi-Tenant Domains

A single Sentry instance serves four domains via K8s Ingress. Each domain maps to a separate Sentry organization, allowing per-org project isolation:

sentry.hanzo.ai -- Hanzo AI services

sentry.lux.network -- Lux blockchain nodes

sentry.pars.network -- Pars network services

sentry.zoo.network -- Zoo Labs experiments

Auth Flow

SSO is via OIDC against hanzo.id:

User visits sentry.hanzo.ai and clicks "SSO Login"

Redirected to hanzo.id for authentication (client app-sentry)

OIDC callback returns to Sentry with identity token

Sentry creates/links user account and grants org membership

Dockerfile Customizations

The self-hosted/Dockerfile in the local clone contains Hanzo-specific changes:

OCI labels with ops@hanzo.ai maintainer and Hanzo branding

References ghcr.io/hanzoai/sentry as image source

Copies self-hosted/sentry.conf.py and self-hosted/config.yml into /etc/sentry/

Entrypoint via tini + gosu for proper signal handling

KMS Secret Sync

Secrets are managed via the KMSSecret CRD that syncs from kms.hanzo.ai:

# kms-secrets.yaml
spec:
  hostAPI: https://kms.hanzo.ai/api
  managedSecretReference:
    secretName: sentry-secrets
    secretNamespace: hanzo
  authentication:
    universalAuth:
      secretsScope:
        projectSlug: hanzo-k8s-epiq
        envSlug: prod
        secretsPath: /sentry

Required secrets: secret-key, db-password, oidc-client-secret, commerce-api-key (optional).

Environment Variables

# Database (PostgreSQL)
SENTRY_POSTGRES_HOST=hanzo-sql.hanzo.svc
SENTRY_DB_NAME=sentry
SENTRY_DB_USER=sentry
SENTRY_DB_PASSWORD=<from KMS>
SENTRY_DB_PORT=5432

# Redis
SENTRY_REDIS_HOST=redis.hanzo.svc
SENTRY_REDIS_PORT=6379

# Core
SENTRY_SECRET_KEY=<from KMS>

# OIDC SSO
SENTRY_OIDC_ENABLED=true
SENTRY_OIDC_ISSUER=https://hanzo.id
SENTRY_OIDC_CLIENT_ID=app-sentry
SENTRY_OIDC_CLIENT_SECRET=<from KMS>

# Observability
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.hanzo.svc:4318
OTEL_SERVICE_NAME=sentry

# Optional: Commerce billing
HANZO_COMMERCE_URL=https://commerce.hanzo.ai
HANZO_COMMERCE_API_KEY=<from KMS>

Local Development

cd ~/work/hanzo/sentry

# Python setup (uses direnv + pyenv, not uv — upstream pattern)
make develop

# Run dependent services (Postgres, Redis, Kafka, etc.)
make run-dependent-services

# Apply database migrations
make apply-migrations

# Start web server
sentry run web

# In another terminal: start worker
sentry run worker

# Run Python tests
make test-python-ci

# Run JavaScript tests
make test-js

Troubleshooting

Issue	Cause	Solution
OIDC login fails	`app-sentry` not configured in IAM	Create OIDC client at hanzo.id with redirect URI `https://sentry.hanzo.ai/auth/sso/`
Redis connection error	Missing `SENTRY_REDIS_HOST`	Ensure `redis.hanzo.svc` is reachable in cluster
`SENTRY_SECRET_KEY is undefined`	KMS sync not running	Verify `sentry-secrets` K8s secret exists in `hanzo` namespace
Migration errors	DB not created	Run `sentry upgrade` to create tables and apply migrations
Worker not processing	Celery not connected to broker	Check Redis connectivity or RabbitMQ config
Health check failing	Sentry not fully started	Wait for `initialDelaySeconds` (60s); check logs for startup errors
`npm ci` fails	Peer dep conflicts	Use `npm install` with `.npmrc` containing `legacy-peer-deps=true`

hanzo/hanzo-o11y.md - Observability stack (OTEL, Prometheus, Grafana)

hanzo/hanzo-id.md - IAM and OIDC SSO configuration

hanzo/hanzo-kms.md - Secret management via KMSSecret CRDs

hanzo/hanzo-universe.md - Production K8s infrastructure and manifests

hanzo/hanzo-database.md - Shared PostgreSQL (hanzo-sql)

Hanzo Sentry - Error Tracking & Performance Monitoring

On this page