Is the ONLY valid arm-vs-arm test: it pairs the two models on the items BOTH…

Is the ONLY valid arm-vs-arm test: it pairs the two models on the items BOTH completed, and answers rescue and damage counts with an exact-McNemar p.

GET /v1/benchmark/compare

Benchmark → · All API references → · Six flows, four surfaces →


Address	`https://api.hanzo.ai/v1/benchmark/compare`
Method	`GET`
Operation	`get_benchmark_compare`
Auth	`Authorization: Bearer $HANZO_API_KEY`

Is the ONLY valid arm-vs-arm test: it pairs the two models on the items BOTH completed, and answers rescue and damage counts with an exact-McNemar p.

Pairing is what prevents the subset artifact — comparing one model's easy subset against another's full run — so n_common, not either arm's own coverage, is the number to read this by.

Both a and b are required. The benchmark defaults to gpqa_diamond.

Request

3 fields.

Field	In	Type	Required	Description
`benchmark`	query	string	—	Benchmark is the catalog id to compare on, defaulting to gpqa_diamond.
`a`	query	string	yes	A is the first model id.
`b`	query	string	yes	B is the second model id.

Response

Status	Body	Meaning
`200`	pairing	ok

200 body — 10 fields.

Field	In	Type	Always	Description
`a`	body	string	—	A is the first model id.
`a_correct`	body	integer	—	ACorrect is how many of those common items A got right.
`b`	body	string	—	B is the second model id.
`b_correct`	body	integer	—	BCorrect is how many of those common items B got right.
`benchmark`	body	string	—	Benchmark is the catalog id the two arms were compared on.
`mcnemar_p`	body	number	—	McnemarP is the two-sided exact binomial p on the discordant pairs.
`n_common`	body	integer	—	NCommon is how many items BOTH arms completed.
`net_a_minus_b`	body	integer	—	NetAMinusB is the two rescue counts subtracted — A's advantage in items.
`rescue_a_over_b`	body	integer	—	RescueAOverB is how many items A got right and B got wrong.
`rescue_b_over_a`	body	integer	—	RescueBOverA is how many items B got right and A got wrong.

Failure carries the platform error shape — see Errors.

Examples

hanzo benchmark compare \
  --a <a> \
  --b <b>

import { Configuration, BenchmarkApi } from 'hanzoai';

const api = new BenchmarkApi(new Configuration({ accessToken: process.env.HANZO_API_KEY }));
const { data } = await api.getBenchmarkCompare({ a: 'a', b: 'b' });

from hanzoai.cloud import ApiClient, Configuration
from hanzoai.cloud.api import BenchmarkApi

client = ApiClient(Configuration(access_token=os.environ["HANZO_API_KEY"]))
result = BenchmarkApi(client).get_benchmark_compare(a='a', b='b')

cfg := cloud.NewConfiguration()
cfg.AddDefaultHeader("Authorization", "Bearer "+os.Getenv("HANZO_API_KEY"))
client := cloud.NewAPIClient(cfg)

resp, _, err := client.BenchmarkAPI.GetBenchmarkCompare(context.Background()).Execute()
if err != nil {
	return err
}

use hanzo_cloud::apis::{configuration::Configuration, benchmark_api};

let mut cfg = Configuration::new();
cfg.bearer_access_token = std::env::var("HANZO_API_KEY").ok();

let result = benchmark_api::get_benchmark_compare(&cfg, Default::default()).await?;

import ai.hanzo.cloud.ApiClient;
import ai.hanzo.cloud.api.BenchmarkApi;

ApiClient client = new ApiClient();
client.setRequestInterceptor(b -> b.header("Authorization", "Bearer " + System.getenv("HANZO_API_KEY")));

var result = new BenchmarkApi(client).getBenchmarkCompare();

curl https://api.hanzo.ai/v1/benchmark/compare?a=%3Ca%3E&b=%3Cb%3E \
  -H "Authorization: Bearer $HANZO_API_KEY"

Tool benchmark, op get_benchmark_compare — POST the JSON-RPC envelope to https://api.hanzo.ai/v1/mcp.

curl -X POST https://api.hanzo.ai/v1/mcp \
  -H "Authorization: Bearer $HANZO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
       "jsonrpc": "2.0",
       "id": 1,
       "method": "tools/call",
       "params": {
         "name": "benchmark",
         "arguments": {
           "op": "get_benchmark_compare",
           "input": {
             "a": "<a>",
             "b": "<b>"
           }
         }
       }
     }'

Benchmark API · All Hanzo APIs · Interactive reference

Is the ONLY valid arm-vs-arm test: it pairs the two models on the items BOTH…

Request

Response

Examples

On this page