Hanzo Console - AI Observability and Prompt Management

Overview

Hanzo Console is the observability and prompt management layer for AI applications. It captures traces, scores, datasets, evaluators and prompts from any LLM application and provides a dashboard for debugging, evaluation, and cost analysis. Everything it shows is readable and writable over the same unified api.hanzo.ai/v1 plane the rest of Hanzo uses — no separate API host, no separate key.

Why Hanzo Console?

• Full trace capture: Every LLM call, tool use, and agent step recorded as a gen_ai span
• Cost attribution: Per-model cost, token and latency breakdown over a window
• Prompt management: A versioned prompt library — creating a prompt whose name already exists appends a version, so nothing changes silently
• Datasets & evals: Build graded example sets, define judges, run them, record scores
• One plane: The same Authorization: Bearer credential as the AI gateway, the same /v1 addresses
• Part of Hanzo ecosystem: Auto-instrumented for Hanzo Chat, Web3, Commerce

When to use

Use this skill when:

• The user wants to trace and debug LLM calls
• The user needs cost analysis across models and providers
• The user wants to manage prompt versions and deployments
• The user needs to build evaluation datasets and run scoring
• The user wants to monitor AI application performance

Hard requirements

1. API key required. Use HANZO_API_KEY, sent as Authorization: Bearer ${HANZO_API_KEY}. Get keys at https://console.hanzo.ai.
2. Never expose keys in user-visible output, logs, or screenshots.
3. Tenancy is server-side. Org and project are minted from the validated credential — never a body field, never a client header. A call with no validated principal is 403, not an empty list, so an empty result never quietly means "wrong key".
4. Scores and traces are telemetry. They need the datastore: a deployment with none wired answers 503 rather than an empty page that would read as "no data".

Preflight checks

Before making any request, silently verify:

• HANZO_API_KEY is set
• The base URL is https://api.hanzo.ai/v1 (https://console.hanzo.ai/v1 serves the identical routes)

Quick reference

One-file quickstart

curl

# The AI overview board: totals, per-model breakdown, latency percentiles
curl "https://api.hanzo.ai/v1/evals/metrics?range=24h" \
  -H "Authorization: Bearer ${HANZO_API_KEY}"

# The traces behind your evaluations
curl "https://api.hanzo.ai/v1/evals/traces?limit=50" \
  -H "Authorization: Bearer ${HANZO_API_KEY}"

# Record a score against a trace
curl -X POST https://api.hanzo.ai/v1/evals/scores \
  -H "Authorization: Bearer ${HANZO_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "quality",
    "value": 0.95,
    "traceId": "trace_abc",
    "comment": "Accurate and concise"
  }'

Python (HTTP)

import os, httpx

api = httpx.Client(
    base_url="https://api.hanzo.ai/v1",
    headers={"Authorization": f"Bearer {os.environ['HANZO_API_KEY']}"},
)

# The board: generations, prompt/completion tokens, cost in cents, errors,
# success rate, a gap-filled series, a per-model breakdown, latency percentiles.
board = api.get("/evals/metrics", params={"range": "7d"}).json()

# Score a trace out of band — human review lands beside the automatic graders.
api.post("/evals/scores", json={
    "name": "quality",
    "value": 0.95,
    "traceId": "trace_abc",
    "comment": "Accurate and concise",
})

Python (model calls through the gateway)

import os
from openai import OpenAI

# Calls made through the Hanzo gateway are recorded as gen_ai spans, which is
# what the LLM-observability views below project over.
client = OpenAI(
    base_url="https://api.hanzo.ai/v1",
    api_key=os.environ["HANZO_API_KEY"],
)

response = client.chat.completions.create(
    model="zen-70b",
    messages=[{"role": "user", "content": "Hello!"}],
)

Endpoint selector

All paths are relative to https://api.hanzo.ai/v1. Routes are keyed by method: on the evals surface a verb that is not listed for a path is a 404, not a 405.

Evaluation traces and scores

Datasets and runs

LLM observability (gen_ai span views)

Reads need the viewer role; the score and annotation writes need editor; the pricing-rule writes need admin.

APM traces

These are reads. There is no POST /o11y/traces — spans arrive from the instrumented call path, not from a client write.

Prompts

Prompt management

A prompt is a name, a template body, and free-form labels / tags. The name is the key: posting a name your org already has appends a new version rather than overwriting one, so the history is real and inspectable.

# Create, or append a version to an existing name
curl -X POST https://api.hanzo.ai/v1/prompts \
  -H "Authorization: Bearer ${HANZO_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "chat-system-prompt",
    "prompt": "You are a helpful assistant for {{user_name}}.",
    "labels": ["production"]
  }'

# Read the current version plus its version history
curl https://api.hanzo.ai/v1/prompts/chat-system-prompt \
  -H "Authorization: Bearer ${HANZO_API_KEY}"

The detail response carries prompt (the current body), version, and versionHistory — version numbers, type and creation time. History is metadata only: past bodies are not served, so a prompt cannot be read out version by version.

Prompt lifecycle

1. Create: author the prompt in the Console dashboard or POST /v1/prompts
2. Version: posting the same name appends a version; nothing is overwritten
3. Label: labels and tags are free-form taxonomy — use them to mark intent
4. Fetch: GET /v1/prompts/{name} returns the current body, so changing a prompt needs no code change and no redeploy
5. Evaluate: score a prompt change against a dataset with /v1/evals/runs

Names must match ^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$, and metrics, catalog and new are reserved because they are sibling routes. A body is capped at 64 KiB and holds template text only — never a secret.

Evaluation and datasets

A dataset is a named set of graded examples. An evaluator is a judge — a model plus the criteria it grades by. A run scores one through the other.

# 1. Create the dataset. Posting the same name edits it; its items are untouched.
curl -X POST https://api.hanzo.ai/v1/evals/datasets \
  -H "Authorization: Bearer ${HANZO_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"name": "chat-eval-v1", "description": "Core product questions"}'

# 2. Add graded examples. Supply "id" to make the write idempotent.
curl -X POST https://api.hanzo.ai/v1/evals/datasets/chat-eval-v1/items \
  -H "Authorization: Bearer ${HANZO_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "input": {"messages": [{"role": "user", "content": "What is Hanzo?"}]},
    "expectedOutput": "Hanzo is an AI infrastructure company...",
    "status": "ACTIVE"
  }'

# 3. Run it. This is synchronous work, not a job id — the summary comes back once
#    every item has been called, traced, judged and scored.
curl -X POST https://api.hanzo.ai/v1/evals/runs \
  -H "Authorization: Bearer ${HANZO_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"dataset": "chat-eval-v1", "model": "zen-70b", "runName": "zen-70b-eval-v1"}'

Things that will bite you if you do not know them:

• Only ACTIVE examples are fed to a run — that is how an example is retired without deleting it. A dataset with none is 422.
• A run executes as you: your own bearer drives the model gateway, so a request without one is 401 rather than a run made under a service identity.
• judge is optional. Omitted, the model under test grades itself against a default correctness criterion, under the score name llm-judge.
• An org may have at most 4 runs in flight; the fifth is 429 rather than queued. A whole run is capped at 10 minutes.
• Deleting a dataset deletes its examples in the same transaction. Scores and runs already recorded against it are telemetry and survive.
• Declare a rubric (POST /v1/evals/rubrics) before recording scores under a name you care about: once one exists its data type is authoritative, and out-of-range values, unlisted labels and non-finite numbers are refused at write time.

MCP Integration

Expose observability as MCP tools:

import { MCPServer, Tool } from '@hanzo/mcp'

const BASE = 'https://api.hanzo.ai/v1'
const auth = { Authorization: `Bearer ${process.env.HANZO_API_KEY}` }

const consoleTools: Tool[] = [
  {
    name: 'console_get_trace',
    description: 'Get trace details for debugging an LLM call',
    parameters: {
      trace_id: { type: 'string', required: true }
    },
    async execute({ trace_id }) {
      const res = await fetch(`${BASE}/o11y/traces/${trace_id}`, { headers: auth })
      return await res.json()
    }
  },
  {
    name: 'console_get_prompt',
    description: 'Fetch a managed prompt by name, with its version history',
    parameters: {
      name: { type: 'string', required: true }
    },
    async execute({ name }) {
      const res = await fetch(`${BASE}/prompts/${name}`, { headers: auth })
      return await res.json()
    }
  },
  {
    name: 'console_cost_summary',
    description: 'Get cost, token and latency breakdown by model over a window',
    parameters: {
      range: { type: 'string', enum: ['24h', '7d', '30d'], default: '24h' }
    },
    async execute({ range }) {
      const res = await fetch(`${BASE}/evals/metrics?range=${range}`, { headers: auth })
      return await res.json()
    }
  }
]

Error handling

Official links

• Dashboard: https://console.hanzo.ai
• Documentation: https://hanzo.ai/docs/console
• Hanzo Python SDK: https://github.com/hanzoai/python-sdk
• Hanzo AI: https://hanzo.ai