Score a model against a dataset you control — one synchronous run grades every item with an LLM judge, records a trace per item, and gives you an average you can compare across changes.

Evals

An eval answers one question: did the change help? You keep a dataset of inputs and expected outputs, run a model over it, and a judge scores each result. Run it again after changing the model, the prompt, or the fine-tune, and the two averages are comparable because the dataset did not move.

Make a dataset

curl -X POST https://api.hanzo.ai/v1/evals/datasets \
  -H "Authorization: Bearer $HANZO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "name": "refund-replies", "description": "Support replies about refunds" }'

name is the key and must match ^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$. POSTing the same name again edits the dataset — it never creates a duplicate, and it leaves the items alone.

Add items

curl -X POST https://api.hanzo.ai/v1/evals/datasets/refund-replies/items \
  -H "Authorization: Bearer $HANZO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "input": { "question": "Can I get a refund 45 days after purchase?" },
    "expectedOutput": "Explains the 30-day window and offers store credit.",
    "status": "ACTIVE"
  }'

input and expectedOutput are raw JSON — a string, an object, whatever your task needs — up to 64 KiB each. status is ACTIVE (the default) or ARCHIVED; a run evaluates active items only, so archiving retires an item without deleting the history that references it.

Supply your own id to make the write idempotent: the same id in the same dataset replaces in place. An unknown dataset answers 404 rather than quietly creating one.

Run it

curl -X POST https://api.hanzo.ai/v1/evals/runs \
  -H "Authorization: Bearer $HANZO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "dataset": "refund-replies",
    "model": "enso-flash",
    "runName": "flash-baseline",
    "limit": 20,
    "judge": {
      "model": "enso",
      "criteria": "The reply must state the 30-day window and stay under 80 words."
    }
  }'

The run is synchronous — it returns the summary, not a job id:

{
  "dataset": "refund-replies",
  "model": "enso-flash",
  "judgeModel": "enso",
  "runName": "flash-baseline",
  "items": 20,
  "scored": 20,
  "avgScore": 0.78,
  "results": [
    { "itemId": "it_9f2c", "traceId": "tr_71ab", "score": 0.9, "output": "..." }
  ]
}

dataset and model are required. limit defaults to 20 and falls back to 20 if you ask for more than 100. Leave runName blank and it is generated from the timestamp.

Defaults worth knowing. With no judge, the model under test grades itself — fine for a smoke test, not for a decision. Name a stronger judge model when the answer matters. The default criteria is that the output should be correct, relevant, and match the expected output.

The judge runs at temperature 0, is given the untrusted content in the user role, must reply as JSON, and its score is clamped to [0, 1]. A reply that cannot be parsed is recorded as an error on that item, never as a fabricated score — which is why scored can be lower than items.

Each item also records a trace named eval:<runName>, with the run name as the session id, so a suspicious score is one click from the actual exchange.

What can go wrong

Status	Means
`401`	No credential. The run executes as you, against your own quota
`404`	That dataset does not belong to your org
`422`	The dataset has no active items
`429`	More than 4 runs already in flight for your org. Runs are refused, never queued

A run has a 10-minute wall clock; items past it come back as errors rather than scores. If nothing scored at all you get 502 — with the same summary body, so you can still see what happened.

Read the results

# Past runs
curl "https://api.hanzo.ai/v1/evals/runs?datasetName=refund-replies" \
  -H "Authorization: Bearer $HANZO_API_KEY"

# Scores, filtered
curl "https://api.hanzo.ai/v1/evals/scores?runName=flash-baseline" \
  -H "Authorization: Bearer $HANZO_API_KEY"

# The traces the run recorded
curl "https://api.hanzo.ai/v1/evals/traces?runName=flash-baseline" \
  -H "Authorization: Bearer $HANZO_API_KEY"

GET /v1/evals/metrics is the aggregate board across your AI traffic — totals, a gap-filled series, per-model breakdown and latency percentiles over 24h, 7d, or 30d.

Scoring by hand

Not every judgement is a model's. POST /v1/evals/scores attaches a score to a trace, a run, a dataset, or a single item:

curl -X POST https://api.hanzo.ai/v1/evals/scores \
  -H "Authorization: Bearer $HANZO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "tone",
    "dataType": "CATEGORICAL",
    "stringValue": "warm",
    "traceId": "tr_71ab",
    "comment": "Right answer, brusque delivery."
  }'

dataType is NUMERIC, CATEGORICAL, or BOOLEAN. Define the shape once as a rubric and every later write is checked against it:

curl -X POST https://api.hanzo.ai/v1/evals/rubrics \
  -H "Authorization: Bearer $HANZO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "name": "tone", "dataType": "CATEGORICAL", "categories": ["warm", "neutral", "brusque"] }'

A declared rubric is authoritative: it overrides whatever dataType a writer claims, a NUMERIC score must fall inside its bounds, a BOOLEAN must be exactly 0 or 1, and a CATEGORICAL stringValue must be one of its categories. Up to 64 categories.

POST /v1/evals/evaluators stores a reusable judge definition — a name, a grading model, criteria, and the scoreName it writes. Runs today take their judge inline, so treat a stored evaluator as the shared definition you paste from.

Experiments — comparing runs against each other
Traces — the per-item execution an eval records
Observations — the generations inside those traces
Prompts — version the prompt you are evaluating
Fine-tuning — prove the tuned model beat the base
API Reference — every endpoint at api.hanzo.ai

Evals

On this page