Evals
Package eval is scoring a model on your own data, with a judge you choose.
Package eval is scoring a model on your own data, with a judge you choose.
| Base URL | https://api.hanzo.ai |
| Operations | 16 |
| Auth | Authorization: Bearer $HANZO_API_KEY |
evals
GET /v1/evals/datasets/{name}/items
The examples in one of your datasets
Lists the examples of ONE dataset as {data:[…]} — the set is named in the path, because this collection only exists inside one. Archived examples are included, so the caller sees the whole set rather than only what a run would use. limit defaults to 100 and is capped at 500.
Requires a validated principal; 403 without one, and the read is filtered on the validated org, so naming another tenant's dataset returns nothing rather than its contents.
| Parameter | In | Type | Required | Description |
|---|---|---|---|---|
name | path | string | yes |
POST /v1/evals/datasets/{name}/items
Add a graded example to one of your datasets
Writes one example — its input, its expectedOutput, free-form metadata and a status — into the dataset named in the path, and answers 201 with it. That dataset MUST already exist for this org: an unknown one is 404, never a silent create, so an example can never be attached to a set the caller does not own.
Supply id to make the write idempotent — re-posting the same id replaces that example in place — or omit it and one is generated. An id that already exists in a DIFFERENT dataset is 409 rather than a move. status is ACTIVE (the default) or ARCHIVED; only ACTIVE examples are fed to a run, which is how an example is retired without deleting it. input and expectedOutput are stored as raw JSON exactly as sent. Requires a validated principal; 403 without one.
| Parameter | In | Type | Required | Description |
|---|---|---|---|---|
name | path | string | yes |
GET /v1/evals/datasets/{name}
One dataset, with how many examples it holds
Returns a single dataset of the caller's org by name, together with its live item count — the one read that answers how big the set actually is. A name this org does not have is 404, which is also what another tenant's dataset looks like from here. Requires a validated principal; 403 without one.
| Parameter | In | Type | Required | Description |
|---|---|---|---|---|
name | path | string | yes |
DELETE /v1/evals/datasets/{name}
Delete a dataset and every example in it
Removes the named dataset of the caller's org AND all of its items, in one transaction, and answers 204. This is not a detach: the examples are gone with the set, so a dataset cannot be resurrected by re-creating the name.
A name this org does not have is 404 — never a silent success — and a name belonging to another tenant is the same 404, because the delete is predicated on the validated org. Requires a validated principal; 403 without one. Runs and scores already recorded against the dataset are telemetry events and are NOT deleted with it.
| Parameter | In | Type | Required | Description |
|---|---|---|---|---|
name | path | string | yes |
GET /v1/evals/datasets
The datasets your org has
Lists the caller org's datasets as {data:[…]}, each with its name, description, metadata and timestamps. limit defaults to 100 and is capped at 500; an unparseable or non-positive value falls back to the default rather than failing.
Requires a validated principal; 403 without one. Every row is filtered on the validated org, so there is no parameter that reaches another tenant's datasets. The items count is NOT populated here — read one dataset to get it.
POST /v1/evals/datasets
Create a dataset, or edit the one with that name
Writes a dataset — the named set of graded examples a run scores a model against — under the caller's org and answers 201 with it. The NAME is the key, not an id: posting a name the org already has updates that dataset's description and metadata and keeps its original creation time, so this is create-or-edit and never a duplicate. Its items are untouched.
Requires a validated principal; 403 without one. The org comes from the validated owner claim, never from a client X-Org-Id, so a dataset can only ever be written under the caller's own tenant. name is required and must match ^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$; a description over 64 KiB is 400.
GET /v1/evals/evaluators
The judges your org has defined
Lists the caller org's evaluators as {data:[…]}, each with its judge model, criteria and the score name it writes under. limit defaults to 100 and is capped at 500. Requires a validated principal; 403 without one, and the listing is filtered on the validated org.
POST /v1/evals/evaluators
Define a judge: a model plus the criteria it grades by
Saves a reusable evaluator for the caller's org — the judge model and the written criteria it grades against — and answers 201 with it. Like a dataset, the NAME is the key: re-posting a name edits that evaluator rather than adding a second one.
scoreName is the name the resulting scores are filed under and defaults to the evaluator's own name; both must match ^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$. Criteria over 64 KiB is 400. Requires a validated principal; 403 without one.
GET /v1/evals/metrics
Your org's AI overview board
Returns the whole observability board for the caller's org over a window: totals (generations, prompt and completion tokens, cost in cents, errors, success rate, distinct models and users), a gap-filled time series, a per-model breakdown with the long tail folded into other, and latency percentiles read from the GenAI spans.
range is 24h (the default), 7d or 30d, and anything else normalises to 24h rather than failing; interval overrides the bucket with hour or day. The window the answer was actually computed over is echoed back, so a client never has to infer it. A platform admin sees the board across ALL orgs; everyone else sees their own.
The board is HONEST-EMPTY where it cannot be computed: with no datastore wired, or under a named project scope the usage ledger does not yet carry, it answers a valid board with zero totals and a flat series rather than a fabricated number or a 500. Requires a validated principal; 403 without one.
GET /v1/evals/rubrics
The score shapes your org has declared
Lists the caller org's rubrics as {data:[…]} — each name's data type, its numeric bounds and its allowed categories. limit defaults to 100 and is capped at 500. Requires a validated principal; 403 without one, and the listing is filtered on the validated org.
POST /v1/evals/rubrics
Declare what a score named X is allowed to be
Defines the shape of one score name for the caller's org — NUMERIC (the default, optionally bounded by minValue/maxValue), CATEGORICAL (a closed set of categories) or BOOLEAN — and answers 201 with it. The NAME is the key, so re-posting a name replaces its rules.
This is the integrity contract, not documentation: once a config exists for a name, every score recorded under that name is checked against it and the config's data type is AUTHORITATIVE — a caller cannot claim a different one. Out-of-range values, unlisted labels and non-finite numbers are refused at write time.
A CATEGORICAL config with no categories is 400, as is a non-finite bound or a minValue above maxValue. Requires a validated principal; 403 without one.
GET /v1/evals/runs
Past runs and how they scored
Lists the caller org's durable run records as {data:[…]} — the dataset and model, the judge model, how many examples were attempted and how many scored, the average score, and when it happened. Narrow to one dataset with datasetName; limit defaults to 100 and is capped at 500.
Requires a validated principal; 403 without one, and rows are filtered on the validated org. These records come from the metastore rather than the datastore, so they are readable on a deployment with no telemetry wired — but a run's traces and scores are not.
POST /v1/evals/runs
Score a dataset through a model and a judge, now
Runs a real evaluation and answers the summary when it is finished — this is synchronous work, not a job id. For each ACTIVE example in the dataset it calls the model under test, records a trace, calls the LLM-as-judge, and records the judge's score with its reasoning. The answer carries the per-item results (item id, trace id, score, output or error) alongside items, scored and avgScore.
dataset and model are required; the dataset must belong to the caller's org (404 otherwise) and must have at least one ACTIVE example (422 otherwise). judge is optional — omitted, the model under test grades itself against a default correctness criterion under the score name llm-judge. limit defaults to 20 and anything above 100 falls back to the default. runName is generated from the clock when omitted.
It runs as YOU: the caller's own Authorization bearer drives the model gateway, so a request without one is 401 rather than a run made anonymously or under a service identity. Only a non-reversible hash of that credential is recorded on the traces.
Bounded and honest about it: an org may have at most 4 runs in flight and the fifth is 429 rather than queued, and the whole run is capped at 10 minutes — items past the deadline come back with an error instead of a score, and scored counts only real successes. A run where NOTHING scored answers 502, not a 200 that looks like an evaluation. A run must be able to persist what it produces, so a deployment with no datastore wired is 503 up front. Requires a validated principal; 403 without one.
GET /v1/evals/scores
Score events, filtered
Lists the caller org's score events as {data:[…]}, narrowed by any of name, runName and traceId; an absent filter simply does not narrow. limit defaults to 100 and is capped at 500.
The org is bound as an authoritative predicate on the query, never taken from a header, so a filter can narrow the caller's own scores but can never widen past them. Requires a validated principal; 403 without one. Scores live in the datastore, so a deployment with none wired answers 503 rather than an empty page that would read as 'no scores'.
POST /v1/evals/scores
Record a score against a trace, a run or an example
Files one score event for the caller's org and answers 201 with it. This is how human review and out-of-band graders land beside the automatic ones: name the score, give it a value (or a stringValue for a categorical label), and attach it to a traceId, a runName, a datasetName/datasetItemId, or any combination.
Scores are validated fail-closed. A value must be FINITE — NaN and Inf are 400 — and if the org has declared a score config for this name, that config decides the type and the value must satisfy it: inside the numeric bounds, or one of the allowed categories. A caller cannot override the declared type by sending a different dataType. Comments are truncated at 2000 characters.
A score is TELEMETRY, not metadata, so it needs the datastore: a deployment with no datastore wired answers 503 rather than accepting a score it cannot persist. Requires a validated principal; 403 without one, and the org is stamped from the validated claim rather than read off the body.
GET /v1/evals/traces
The traces behind your evaluations
Lists the caller org's traces as {data:[…]} — one per model call an evaluation made, carrying its input, output, model and timing — narrowed by any of sessionId, runName and datasetName. limit defaults to 100 and is capped at 500.
Scoped by org AND by project: the project is the caller's server-minted scope, not a parameter, so it cannot be widened by asking. Requires a validated principal; 403 without one. Traces live in the datastore, so a deployment with none wired answers 503 rather than an empty page.
How is this guide?