Runs a real evaluation and answers the summary when it is finished — this is…

Runs a real evaluation and answers the summary when it is finished — this is synchronous work, not a job id.

POST /v1/eval/runs

Eval → · All API references → · Six flows, four surfaces →


Address	`https://api.hanzo.ai/v1/eval/runs`
Method	`POST`
Operation	`post_eval_runs`
Auth	`Authorization: Bearer $HANZO_API_KEY`

Runs a real evaluation and answers the summary when it is finished — this is synchronous work, not a job id.

For each ACTIVE example in the dataset it calls the model under test, records a trace, calls the LLM-as-judge, and records the judge's score with its reasoning. The answer carries the per-item results (item id, trace id, score, output or error) alongside items, scored and avgScore.

The dataset must belong to the caller's org (404 otherwise) and must have at least one ACTIVE example (422 otherwise).

It runs as YOU: the caller's own Authorization bearer drives the model gateway, so a request without one is 401 rather than a run made anonymously or under a service identity. Only a non-reversible hash of that credential is recorded on the traces.

Bounded and honest about it: an org may have at most 4 runs in flight and the fifth is 429 rather than queued, and the whole run is capped at 10 minutes — examples past the deadline come back with an error instead of a score, and scored counts only real successes. A run where NOTHING scored answers 502, not a 200 that looks like an evaluation. A run must be able to persist what it produces, so a deployment with no datastore wired is 503 up front. Requires a validated principal; 403 without one.

Request

9 fields, body application/json (required).

Field	In	Type	Required	Description
`Authorization`	header	string	—
`dataset`	body	string	yes	Dataset is the set to score, which must belong to the caller's org and hold at least one ACTIVE example.
`judge`	body	judgeSpec	—
`judge.criteria`	body	string	—	Criteria is the standard the judge applies, defaulting to a correctness criterion.
`judge.model`	body	string	—	Model is the model that grades.
`judge.name`	body	string	—	Name is the score name the judge's grades are filed under, "llm-judge" by default.
`limit`	body	integer	—	Limit caps how many examples this run scores.
`model`	body	string	yes	Model is the model under test.
`runName`	body	string	—	RunName labels the run and is generated from the clock when omitted.

Response

Status	Body	Meaning
`200`	runSummary	ok
`502`	runSummary	bad gateway

200 body — 13 fields.

Field	In	Type	Always	Description
`avgScore`	body	number	—	AvgScore is the mean over the scored examples, 0 when none scored.
`dataset`	body	string	—	Dataset is the set that was scored.
`items`	body	integer	—	Items is how many examples the run attempted.
`judgeModel`	body	string	—	JudgeModel is the model that graded.
`model`	body	string	—	Model is the model under test.
`results`	body	itemResult[]	—	Results is one row per attempted example.
`results[].error`	body	string	—	Error is why this example produced no score — the model, the judge, or the run's deadline.
`results[].itemId`	body	string	—	ItemID is the example that was scored.
`results[].output`	body	string	—	Output is what the model under test answered, truncated at 2000 characters.
`results[].score`	body	number	—	Score is the judge's grade.
`results[].traceId`	body	string	—	TraceID is the model call this result came from.
`runName`	body	string	—	RunName is the run's label, which scores and traces are filed under.
`scored`	body	integer	—	Scored is how many produced a real score.

Failure carries the platform error shape — see Errors.

Examples

hanzo evals runs create \
  --dataset <dataset> \
  --model <model>

import { Configuration, EvalApi } from 'hanzoai';

const api = new EvalApi(new Configuration({ accessToken: process.env.HANZO_API_KEY }));
const { data } = await api.postEvalRuns({ dataset: "<dataset>", model: "<model>" });

from hanzoai.cloud import ApiClient, Configuration
from hanzoai.cloud.api import EvalApi

client = ApiClient(Configuration(access_token=os.environ["HANZO_API_KEY"]))
result = EvalApi(client).post_eval_runs(dataset="<dataset>", model="<model>")

cfg := cloud.NewConfiguration()
cfg.AddDefaultHeader("Authorization", "Bearer "+os.Getenv("HANZO_API_KEY"))
client := cloud.NewAPIClient(cfg)

resp, _, err := client.EvalAPI.PostEvalRuns(context.Background()).Execute()
if err != nil {
	return err
}

use hanzo_cloud::apis::{configuration::Configuration, eval_api};

let mut cfg = Configuration::new();
cfg.bearer_access_token = std::env::var("HANZO_API_KEY").ok();

let result = eval_api::post_eval_runs(&cfg, Default::default()).await?;

import ai.hanzo.cloud.ApiClient;
import ai.hanzo.cloud.api.EvalApi;

ApiClient client = new ApiClient();
client.setRequestInterceptor(b -> b.header("Authorization", "Bearer " + System.getenv("HANZO_API_KEY")));

var result = new EvalApi(client).postEvalRuns();

The method above is the one at the current release of the document. [email protected] (npm) and [email protected] (PyPI) were generated from an earlier release, where this operation carried a different id, so it spells the method differently — regenerating the clients is what makes the two agree. SDKs →

curl -X POST https://api.hanzo.ai/v1/eval/runs \
  -H "Authorization: Bearer $HANZO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
       "dataset": "<dataset>",
       "model": "<model>"
     }'

The door reaches eval through the evals tool, which names its 16 operations with its own verbs — this one among them, under a name only the door declares. describe explains any of them:

curl -X POST https://api.hanzo.ai/v1/mcp \
  -H "Content-Type: application/json" \
  -d '{
       "jsonrpc": "2.0",
       "id": 1,
       "method": "tools/call",
       "params": {
         "name": "describe",
         "arguments": {
           "op": "list_eval_datasets"
         }
       }
     }'

Eval API · All Hanzo APIs · Interactive reference

Runs a real evaluation and answers the summary when it is finished — this is…

Request

Response

Examples

On this page