Hanzo
Services

Hanzo Console

LLM engineering platform for observability, prompt management, evaluations, datasets, cost tracking, and session replay with distributed tracing.

Hanzo Console

Hanzo Console is the LLM engineering platform at the center of the Hanzo AI stack. It provides full-lifecycle observability for AI applications: distributed tracing, prompt management, evaluation pipelines, dataset curation, cost tracking, session replay, and A/B testing. All Hanzo services emit telemetry to Console, giving operators and developers a single pane of glass over every LLM call, agent step, and tool invocation.

Dashboard: console.hanzo.ai API: api.hanzo.ai/v1/console/*

Features

  • Distributed Tracing -- OpenTelemetry-compatible trace ingestion with span-level detail for LLM calls, retrievals, tool use, and agent chains
  • Prompt Management -- Version-controlled prompt templates with variable interpolation, rollback, and deployment slots
  • Evaluation Pipelines -- Automated scoring with built-in and custom evaluators (LLM-as-judge, regex, cosine similarity, human review)
  • Dataset Management -- Curate, version, and export datasets for fine-tuning and evaluation from production traces
  • Cost Tracking -- Per-model, per-project, and per-user cost attribution with budget alerts
  • Session Replay -- Reconstruct full user sessions across multiple traces and generations
  • A/B Testing -- Split traffic across prompt variants and compare metrics side-by-side
  • Multi-Org Support -- Organization isolation via Hanzo IAM SSO; switch between orgs in one click
  • OpenAI SDK Compatible -- Drop-in wrapper; change one line to start tracing existing OpenAI or Hanzo SDK calls

Architecture

┌──────────────────────────────────────────────────────────────────┐
│                     console.hanzo.ai (Next.js 15, :3001)         │
├──────────────────────────────────────────────────────────────────┤
│  Traces Explorer | Prompts Manager | Evals Pipeline | Datasets  │
│                          │                                       │
│                    Console API                                   │
│            ┌─────────────┼─────────────┐                        │
│      PostgreSQL     ClickHouse      Redis ──► Worker            │
│      (metadata)     (analytics)     (queues)  (BullMQ 27+)     │
└──────────────────────────────────────────────────────────────────┘

Ingestion:  Your App ──► SDK/HTTP ──► api.hanzo.ai/v1/console/* ──► ClickHouse
ComponentImagePortPurpose
Webhanzoai/console:latest3001Next.js 15 dashboard and API
Workerghcr.io/hanzoai/console-worker:latest--BullMQ processor (27+ queues)
ClickHouseIn-cluster9000Analytics storage (traces, scores, costs)
PostgreSQLIn-cluster5432Metadata (prompts, datasets, projects, users)
RedisIn-cluster6379Job queues, caching, pub/sub

Quick Start

Get an API Key

Sign in at console.hanzo.ai with Hanzo IAM SSO. Navigate to Settings > API Keys and create a new key. Keys are scoped to a project and prefixed with hk-.

Send Your First Trace

curl -X POST https://api.hanzo.ai/v1/console/ingestion \
  -H "Authorization: Bearer hk-your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "batch": [
      {
        "type": "trace-create",
        "id": "trace-001",
        "timestamp": "2026-02-22T00:00:00Z",
        "body": {
          "name": "my-first-trace",
          "input": {"query": "What is Hanzo Console?"},
          "output": {"answer": "An LLM engineering platform."},
          "metadata": {"env": "production"}
        }
      },
      {
        "type": "generation-create",
        "id": "gen-001",
        "timestamp": "2026-02-22T00:00:01Z",
        "body": {
          "traceId": "trace-001",
          "name": "llm-call",
          "model": "anthropic-claude-haiku-4.5",
          "input": [{"role": "user", "content": "What is Hanzo Console?"}],
          "output": {"role": "assistant", "content": "An LLM engineering platform."},
          "usage": {"inputTokens": 12, "outputTokens": 8}
        }
      }
    ]
  }'

View in Dashboard

Open console.hanzo.ai, select your project, and click Traces. Your trace appears with the generation nested inside, showing latency, token usage, and cost.

┌──────────────────────────────────────────────────────────────────┐
│  Hanzo Console             [hanzo ▾]  [prod ▾]  Last 24h ▾  ⟳   │
├────────┬─────────────────────────────────────────────────────────┤
│        │ Traces: 48,291    Generations: 127,843   Cost: $312.47  │
│ Traces │ Avg Latency: 847ms  Error Rate: 0.3%    Scores: 4.2/5  │
│ Genera-│                                                         │
│  tions │ Trace Volume  ▁▂▃▅▆▇█▇▆▅▃▂▁▁▂▃▅▆▇█▇▆▅▃▂▁             │
│ Scores │                                                         │
│ Prompts│ Model Usage          │ Cost by Model                    │
│ Datasets│ claude-haiku-4.5 47% │ claude-haiku-4.5    $147.20     │
│ Evals  │ qwen3-32b        28% │ qwen3-32b           $89.60      │
│ Users  │ gpt-5-nano       15% │ gpt-5-nano          $46.87      │
│ Sessions│ other            10% │ other               $28.80      │
│ Settings│                      │                                  │
│        │ Recent Traces                                           │
│        │ trace-a1b2 rag-pipeline  1.2s $0.004 4.5/5  2m ago    │
│        │ trace-c3d4 chat-session  0.8s $0.002 4.8/5  3m ago    │
│        │ trace-e5f6 agent-step    3.1s $0.012 2.1/5  5m ago    │
└────────┴─────────────────────────────────────────────────────────┘

SDK Integration

Python

from hanzoai import Hanzo

client = Hanzo(api_key="hk-your-api-key", trace=True)

# Every call is traced automatically
response = client.chat.completions.create(
    model="anthropic-claude-haiku-4.5",
    messages=[{"role": "user", "content": "Explain distributed tracing."}],
    metadata={"session_id": "sess-abc", "user_id": "user-42"},
)

Decorator-Based Tracing

from hanzoai.console import observe

@observe()
def my_pipeline(query: str) -> str:
    docs = retrieve(query)       # span: retrieval
    answer = generate(docs)      # span: generation
    return answer

@observe(name="generation", capture_input=True, capture_output=True)
def generate(docs: list) -> str:
    return client.chat.completions.create(
        model="alibaba-qwen3-32b",
        messages=[{"role": "user", "content": str(docs)}],
    ).choices[0].message.content

Prompt Management

from hanzoai.console import get_prompt

prompt = get_prompt(name="qa-system", version=3)
compiled = prompt.compile(context="Hanzo docs", question="What is Console?")

response = client.chat.completions.create(
    model="anthropic-claude-haiku-4.5",
    messages=compiled,
    hanzo_prompt=prompt,  # links generation to prompt version
)

JavaScript / TypeScript

import Hanzo from '@hanzo/ai'

const client = new Hanzo({ apiKey: 'hk-your-api-key', trace: true })

const response = await client.chat.completions.create({
  model: 'anthropic-claude-haiku-4.5',
  messages: [{ role: 'user', content: 'Explain distributed tracing.' }],
  metadata: { sessionId: 'sess-abc', userId: 'user-42' },
})

Manual Spans

import { ConsoleClient } from '@hanzo/ai/console'

const console = new ConsoleClient({ publicKey: 'hk-your-api-key' })
const trace = console.trace({ name: 'rag-pipeline' })

const retrieval = trace.span({ name: 'retrieval', input: { query } })
const docs = await vectorSearch(query)
retrieval.end({ output: docs })

const generation = trace.generation({
  name: 'llm-call',
  model: 'alibaba-qwen3-32b',
  input: messages,
})
const result = await llmCall(messages)
generation.end({ output: result, usage: { inputTokens: 120, outputTokens: 45 } })

OpenAI Drop-In

from openai import OpenAI
from hanzoai.console import wrap_openai

openai_client = wrap_openai(OpenAI(
    api_key="your-hanzo-key",
    base_url="https://api.hanzo.ai/v1",
))

# All calls are now traced to Hanzo Console
response = openai_client.chat.completions.create(
    model="openai-gpt-5-nano",
    messages=[{"role": "user", "content": "Hello!"}],
)

API Reference

All endpoints are under api.hanzo.ai/v1/console/.

Ingestion

MethodPathDescription
POST/ingestionBatch ingest traces, spans, generations, scores, events

Batch event types: trace-create, span-create, span-update, generation-create, generation-update, score-create, event-create.

Traces

MethodPathDescription
GET/tracesList traces with filtering and pagination
GET/traces/:idGet trace with all nested spans

Prompts

MethodPathDescription
GET/promptsList all prompts
GET/prompts/:nameGet prompt by name (latest or specific version)
POST/promptsCreate or update a prompt
curl -X POST https://api.hanzo.ai/v1/console/prompts \
  -H "Authorization: Bearer hk-your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "qa-system",
    "prompt": "Answer based on context.\n\nContext: {{context}}\nQuestion: {{question}}",
    "config": {"model": "anthropic-claude-haiku-4.5", "temperature": 0.2},
    "labels": ["production"]
  }'

Scores

MethodPathDescription
POST/scoresAttach a score to a trace or generation
GET/scoresList scores with filtering

Datasets

MethodPathDescription
GET/datasetsList datasets
POST/datasetsCreate a dataset
POST/datasets/:id/itemsAdd items to a dataset
POST/datasets/:id/runsExecute an evaluation run

Sessions and Metrics

MethodPathDescription
GET/sessionsList sessions
GET/sessions/:idGet session with all traces
GET/metrics/dailyDaily aggregated metrics (cost, latency, volume)
GET/metrics/usageToken usage breakdown by model

Evaluators

EvaluatorTypeDescription
llm-as-judgeLLMConfigurable LLM scores outputs on custom criteria
cosine-similarityNumericEmbedding similarity between output and reference
containsBooleanChecks output contains expected substrings
regex-matchBooleanMatches output against regex patterns
json-validityBooleanValidates JSON structure
customFunctionUser-defined scoring via webhook

Configuration

Required Environment Variables

VariableDescription
DATABASE_URLPostgreSQL connection string
CLICKHOUSE_URLClickHouse connection string
REDIS_URLRedis connection string
NEXTAUTH_URLConsole public URL (https://console.hanzo.ai)
NEXTAUTH_SECRETNextAuth session encryption key
HANZO_IAM_CLIENT_IDIAM app client ID (hanzo-console-client-id)
HANZO_IAM_CLIENT_SECRETIAM app client secret
HANZO_IAM_ISSUERIAM OIDC issuer URL (default: https://hanzo.id)
SALTAPI key hashing salt

Optional: HANZO_S3_EVENT_UPLOAD_ENDPOINT, HANZO_S3_EVENT_UPLOAD_BUCKET (hanzo-events), HANZO_S3_MEDIA_UPLOAD_BUCKET (hanzo-media), HANZO_INIT_ORG_IDS, HANZO_INIT_USER_EMAIL, HANZO_INIT_PROJECT_ORG_ID, AGENTS_API_URL.

ClickHouse Migrations

34 migrations managed by golang-migrate. Database must be named console (not hanzo).

kubectl port-forward svc/clickhouse 9000:9000 -n hanzo
migrate -path ./migrations -database "clickhouse://localhost:9000/console" up

Kubernetes

Console runs as two deployments on hanzo-k8s: console (2 replicas, hanzoai/console:latest, port 3001, health at /api/public/health) and console-worker (1 replica, ghcr.io/hanzoai/console-worker:latest, imagePullSecrets: ghcr-secret).

Worker processes 27+ BullMQ queues: trace-upsert, generation-upsert, score-upsert, event-upsert, dataset-run, prompt-cache-invalidation, cost-calculation, session-aggregation, export, and more.

DNS: console.hanzo.ai and api.hanzo.ai both resolve to 24.199.76.156 (hanzo-k8s LB, Cloudflare proxied). The gateway routes /console/* to the console service.

How is this guide?

Last updated on

On this page