Eval
Package eval is scoring a model on your own data, with a judge you choose.
Package eval is scoring a model on your own data, with a judge you choose.
| Base URL | https://api.hanzo.ai |
| Operations | 16 |
| Auth | Authorization: Bearer $HANZO_API_KEY |
Specification
Specification pending — no HIP in hanzoai/hips declares capability: eval yet. What this capability serves is below, from the API document; what it is — the store it owns, how it meters, what it publishes — is written as a HIP under HIP-0139.
Four surfaces
| Surface | Reaches this capability as | Coverage |
|---|---|---|
| REST | eval at its own prefix | 16 operations |
| CLI | hanzo evals … | 16 of 16 |
| SDK | — | no published client declares one yet — regenerating the clients is what adds them |
| MCP | tool evals on https://api.hanzo.ai/v1/mcp | 16 operations, 0 under the document's own id — ask describe for the rest |
Quickstart
export HANZO_API_KEY=sk-... # console.hanzo.ai → API keysThen the first call — a read that needs nothing but the key. GET /v1/eval/runs, operation get_eval_runs:
hanzo evals runs getimport { Configuration, EvalApi } from 'hanzoai';
const api = new EvalApi(new Configuration({ accessToken: process.env.HANZO_API_KEY }));
const { data } = await api.getEvalRuns();from hanzoai.cloud import ApiClient, Configuration
from hanzoai.cloud.api import EvalApi
client = ApiClient(Configuration(access_token=os.environ["HANZO_API_KEY"]))
result = EvalApi(client).get_eval_runs()cfg := cloud.NewConfiguration()
cfg.AddDefaultHeader("Authorization", "Bearer "+os.Getenv("HANZO_API_KEY"))
client := cloud.NewAPIClient(cfg)
resp, _, err := client.EvalAPI.GetEvalRuns(context.Background()).Execute()
if err != nil {
return err
}use hanzo_cloud::apis::{configuration::Configuration, eval_api};
let mut cfg = Configuration::new();
cfg.bearer_access_token = std::env::var("HANZO_API_KEY").ok();
let result = eval_api::get_eval_runs(&cfg, Default::default()).await?;import ai.hanzo.cloud.ApiClient;
import ai.hanzo.cloud.api.EvalApi;
ApiClient client = new ApiClient();
client.setRequestInterceptor(b -> b.header("Authorization", "Bearer " + System.getenv("HANZO_API_KEY")));
var result = new EvalApi(client).getEvalRuns();The method above is the one at the current release of the document. [email protected] (npm) and [email protected] (PyPI) were generated from an earlier release, where this operation carried a different id, so it spells the method differently — regenerating the clients is what makes the two agree. SDKs →
curl https://api.hanzo.ai/v1/eval/runs \
-H "Authorization: Bearer $HANZO_API_KEY"The door reaches eval through the evals tool, which names its 16 operations with its own verbs — this one among them, under a name only the door declares. describe explains any of them:
curl -X POST https://api.hanzo.ai/v1/mcp \
-H "Content-Type: application/json" \
-d '{
"jsonrpc": "2.0",
"id": 1,
"method": "tools/call",
"params": {
"name": "describe",
"arguments": {
"op": "list_eval_datasets"
}
}
}'Answers 200 with object — ok.
Endpoints
| Endpoint | What it does |
|---|---|
GET /v1/eval/datasets/{name}/items | Is the examples in one of your datasets — the set is named in the path, because this collection only exists inside one. |
POST /v1/eval/datasets/{name}/items | Writes one graded example — its input, its expected output, free-form metadata and a status — into the dataset named in the path, and answers 201… |
GET /v1/eval/datasets/{name} | Returns one dataset of the caller's org by name, together with its live item count — the one read that answers how big the set actually is. |
DELETE /v1/eval/datasets/{name} | Removes the named dataset of the caller's org AND all of its examples, in one transaction. |
GET /v1/eval/datasets | Is the datasets your org has, each with its name, description, metadata and timestamps. |
POST /v1/eval/datasets | Writes a dataset — the named set of graded examples a run scores a model against — under the caller's org and answers 201 with it. |
GET /v1/eval/evaluators | Is the judges your org has defined, each with its judge model, criteria and the score name it writes under. |
POST /v1/eval/evaluators | Saves a reusable judge for the caller's org — the judge model and the written criteria it grades against — and answers 201 with it. |
GET /v1/eval/metrics | Is your org's AI overview board over a window: totals (generations, prompt and completion tokens, cost in cents, errors, success rate, distinct… |
GET /v1/eval/rubrics | Is the score shapes your org has declared — each name's data type, its numeric bounds and its allowed categories. |
POST /v1/eval/rubrics | Defines the shape of one score name for the caller's org and answers 201 with it. |
GET /v1/eval/runs | Is your past runs and how they scored — the dataset and model, the judge model, how many examples were attempted and how many scored, the average… |
POST /v1/eval/runs | Runs a real evaluation and answers the summary when it is finished — this is synchronous work, not a job id. |
GET /v1/eval/scores | Is the score events your org has recorded, narrowed by any of name, runName and traceId. |
POST /v1/eval/scores | Files one score event for the caller's org and answers 201 with it. |
GET /v1/eval/traces | Is the traces behind your evaluations — one per model call an evaluation made, carrying its input, output, model and timing — narrowed by any of… |
How is this guide?