Eval

Scoring a model on your own data, with a judge you choose.

Also for this capability: API · CLI · SDKs

Scoring a model on your own data, with a judge you choose.

Base URL	`https://api.hanzo.ai`
Operations	16
Auth	`Authorization: Bearer $HANZO_API_KEY`

Specification

Specification pending — no HIP in hanzoai/hips declares capability: eval yet. What this capability serves is below, from the API document; what it is — the store it owns, how it meters, what it publishes — is written as a HIP under HIP-0139.

Four surfaces

Surface	Reaches this capability as	Coverage
REST	`eval` at its own prefix	16 operations
CLI	`hanzo eval …`	16 of 16
SDK	`EvalApi` in every published client	16 methods
MCP	tool `eval` on `https://api.hanzo.ai/v1/mcp`	16 operations, 0 under the document's own id — ask `describe` for the rest

Quickstart

export HANZO_API_KEY=sk-...   # console.hanzo.ai → API keys

Then the first call — a read that needs nothing but the key. GET /v1/eval/runs, operation get_eval_runs:

CLI

SDK

HTTP

MCP

hanzo eval runs get

.ts

.py

.go

.rs

.java

import { Configuration, EvalApi } from 'hanzoai';

const api = new EvalApi(new Configuration({ accessToken: process.env.HANZO_API_KEY }));
const { data } = await api.getEvalRuns();

from hanzoai.cloud import ApiClient, Configuration
from hanzoai.cloud.api import EvalApi

client = ApiClient(Configuration(access_token=os.environ["HANZO_API_KEY"]))
result = EvalApi(client).get_eval_runs()

cfg := hanzoai.NewConfiguration()
cfg.AddDefaultHeader("Authorization", "Bearer "+os.Getenv("HANZO_API_KEY"))
client := hanzoai.NewAPIClient(cfg)

resp, _, err := client.EvalAPI.GetEvalRuns(context.Background()).Execute()
if err != nil {
	return err
}

use hanzo_client::apis::{configuration::Configuration, eval_api};

let mut cfg = Configuration::new();
cfg.bearer_access_token = std::env::var("HANZO_API_KEY").ok();

let result = eval_api::get_eval_runs(&cfg, Default::default()).await?;

import ai.hanzo.cloud.ApiClient;
import ai.hanzo.cloud.api.EvalApi;

ApiClient client = new ApiClient();
client.setBearerToken(System.getenv("HANZO_API_KEY"));

var result = new EvalApi(client).getEvalRuns();

curl https://api.hanzo.ai/v1/eval/runs \
  -H "Authorization: Bearer $HANZO_API_KEY"

MCP reaches eval through the eval tool, which names its 16 operations with its own verbs — this one among them, under a name only MCP declares. describe explains any of them:

curl -X POST https://api.hanzo.ai/v1/mcp \
  -H "Content-Type: application/json" \
  -d '{
       "jsonrpc": "2.0",
       "id": 1,
       "method": "tools/call",
       "params": {
         "name": "describe",
         "arguments": {
           "op": "list_eval_datasets"
         }
       }
     }'

Answers 200 with object — ok.

Endpoints

Endpoint	What it does
`GET /v1/eval/datasets/{name}/items`	The examples in one of your datasets — the set is named in the path, because this collection only exists inside one.
`POST /v1/eval/datasets/{name}/items`	Writes one graded example — its input, its expected output, free-form metadata and a status — into the dataset named in the path, and answers 201 with it.
`GET /v1/eval/datasets/{name}`	Returns one dataset of the caller's org by name, together with its live item count — the one read that answers how big the set actually is.
`DELETE /v1/eval/datasets/{name}`	Removes the named dataset of the caller's org AND all of its examples, in one transaction.
`GET /v1/eval/datasets`	The datasets your org has, each with its name, description, metadata and timestamps.
`POST /v1/eval/datasets`	Writes a dataset — the named set of graded examples a run scores a model against — under the caller's org and answers 201 with it.
`GET /v1/eval/evaluators`	The judges your org has defined, each with its judge model, criteria and the score name it writes under.
`POST /v1/eval/evaluators`	Saves a reusable judge for the caller's org — the judge model and the written criteria it grades against — and answers 201 with it.
`GET /v1/eval/metrics`	Your org's AI overview board over a window: totals (generations, prompt and completion tokens, cost in cents, errors, success rate, distinct models and users), a gap-filled time series, a per-model breakdown with the long tail folded into "other", and latency percentiles read from the GenAI spans.
`GET /v1/eval/rubrics`	The score shapes your org has declared — each name's data type, its numeric bounds and its allowed categories.
`POST /v1/eval/rubrics`	Defines the shape of one score name for the caller's org and answers 201 with it.
`GET /v1/eval/runs`	Your past runs and how they scored — the dataset and model, the judge model, how many examples were attempted and how many scored, the average score, and when it happened.
`POST /v1/eval/runs`	Runs a real evaluation and answers the summary when it is finished — this is synchronous work, not a job id.
`GET /v1/eval/scores`	The score events your org has recorded, narrowed by any of name, runName and traceId.
`POST /v1/eval/scores`	Files one score event for the caller's org and answers 201 with it.
`GET /v1/eval/traces`	The traces behind your evaluations — one per model call an evaluation made, carrying its input, output, model and timing — narrowed by any of sessionId, runName and datasetName.

All Hanzo APIs · Interactive reference

Was this page useful?

PreviousSkills NextBenchmark