NVIDIA NeMo

NeMo is not one product. It is a set of libraries and containers that between them cover a model's life: open models (Nemotron), data curation and synthesis (Curator, Data Designer, Anonymizer, Safe Synthesizer), post-training (RL, Gym, Customizer, Megatron-Bridge), evaluation (Evaluator, Auditor), runtime safety (Guardrails), inference (NIM) and agent telemetry (Relay, Agent Toolkit). Some of those are services you run; some are libraries you import. The split matters here, because the two halves migrate differently.

The services have a capability that answers them. /v1/ai (272 operations) serves the models at the addresses a NIM or integrate.api.nvidia.com client already calls, and every call it serves is already a gen_ai trace in /v1/o11y (381), which is the half of Relay that records. /v1/eval (16) and /v1/benchmark (9) are Evaluator's two modes; /v1/agent (37) is the Agent Toolkit's workflow with its tools resolved from /v1/tool (19).

The libraries have no counterpart and need none. Curator, Data Designer, Anonymizer, Safe Synthesizer and NeMo RL take an OpenAI-compatible provider and your own GPUs; they keep both, and the provider becomes api.hanzo.ai. Nothing on this page asks you to stop using a NeMo library that works.

Start here

Mint a key, point the client that called Nemotron at a different host, then run a benchmark against what it now calls.

# 1. mint a key — sk- belongs on a server, pk- is safe in a browser
curl -sS -X POST https://api.hanzo.ai/v1/account/keys \
  -H "Authorization: Bearer $HANZO_SESSION" \
  -H 'Content-Type: application/json' \
  -d '{"type":"secret"}'

# 2. the call a NIM client already makes, at a different host and model id
curl -sS -X POST https://api.hanzo.ai/v1/chat/completions \
  -H "Authorization: Bearer $HANZO_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model":"zen5","messages":[{"role":"user","content":"summarise Q3"}]}'

# 3. Evaluator's benchmark mode, as an admission rather than a local run
curl -sS -X POST https://api.hanzo.ai/v1/benchmark/runs \
  -H "Authorization: Bearer $HANZO_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model":"zen5","benchmarks":["mmlu","gsm8k"]}'

Step 2 is the request your code already sends. NIM and the NVIDIA API are OpenAI-shaped, so the body is unchanged and the edit is the base URL, the key and the model name; GET /v1/models publishes the names, zen5 among them. Step 3 answers 202 with a receipt, and the numbers appear on GET /v1/benchmark/leaderboard as the harness completes them — the same benchmarks nel eval run --bench would run on your machine, run for you.

Core capabilities

Nouns

Five tables, one per NeMo concern.

Models and inference

Telemetry and control

Relay does two things: it emits OpenTelemetry GenAI spans for LLM and tool calls, and it runs middleware around them. The first is already done for every call /v1/ai serves. The second is a property of the agent definition.

Guardrails

Evaluation

Data and training

The call

One guarded, observed inference in NeMo is three components: a NIM serving the model, Guardrails in front of it with a config.yml, and Relay around the client so the call is a span.

# config.yml — Guardrails
models:
  - type: main
    engine: nim
    model: meta/llama-3.3-70b-instruct
  - type: content_safety
    engine: nim
    model: nvidia/llama-3.1-nemoguard-8b-content-safety
rails:
  input:
    flows: [content safety check input $model=content_safety]
  output:
    flows: [content safety check output $model=content_safety]

# the client, wrapped so Relay emits the gen_ai span
nemo-relay run -- python app.py     # app.py calls http://localhost:8000/v1 through the rails

Hanzo. The safety check is a model call, the inference is a model call, and both are traces because they were served:

curl -sS -X POST https://api.hanzo.ai/v1/chat/completions \
  -H "Authorization: Bearer $HANZO_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model":"zen-guard","messages":[{"role":"user","content":"summarise Q3"}]}'

curl -sS -X POST https://api.hanzo.ai/v1/chat/completions \
  -H "Authorization: Bearer $HANZO_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model":"zen5","messages":[{"role":"user","content":"summarise Q3"}]}'

curl -sS "https://api.hanzo.ai/v1/o11y/llm/traces?limit=20" \
  -H "Authorization: Bearer $HANZO_API_KEY"

The third command is a read. Nothing was instrumented and no exporter was configured; the two calls are in the trace list because the model was served on this side of the request. Where the check should block, your code reads the classifier's answer and does not make the second call — which is what the input rail did, with the decision in a flow file instead of in the caller. With guard-proxy --upstream https://api.hanzo.ai --port 8080 in front and OPENAI_BASE_URL=http://localhost:8080, the PII, injection and content stages run on both calls and the caller does not change.

Evaluation separates the same way. Evaluator on your machine:

export NVIDIA_API_KEY=…
nel eval run --bench mmlu \
  --model-url https://integrate.api.nvidia.com/v1 \
  --model-id nvidia/nemotron-3-super-120b-a12b \
  --api-key $NVIDIA_API_KEY --repeats 3 --max-problems 100
nel eval report ./eval_results/ -f markdown -o report.md

Here, for the standard benchmarks, an admission and a read:

curl -sS -X POST https://api.hanzo.ai/v1/benchmark/runs \
  -H "Authorization: Bearer $HANZO_API_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"zen5","benchmarks":["mmlu"]}'

curl -sS "https://api.hanzo.ai/v1/benchmark/leaderboard" \
  -H "Authorization: Bearer $HANZO_API_KEY"

And for your own graded set, three commands from nothing to a scored run:

curl -sS -X POST https://api.hanzo.ai/v1/eval/datasets \
  -H "Authorization: Bearer $HANZO_API_KEY" -H 'Content-Type: application/json' \
  -d '{"name":"support-qa","description":"answers we grade each release against"}'

curl -sS -X POST https://api.hanzo.ai/v1/eval/datasets/support-qa/items \
  -H "Authorization: Bearer $HANZO_API_KEY" -H 'Content-Type: application/json' \
  -d '{"id":"rotate-key","input":{"question":"how do I rotate a key?"},
       "expectedOutput":{"answer":"mint a new one, deploy it, revoke the old one"}}'

curl -sS -X POST https://api.hanzo.ai/v1/eval/runs \
  -H "Authorization: Bearer $HANZO_API_KEY" -H 'Content-Type: application/json' \
  -d '{"dataset":"support-qa","model":"zen5","runName":"rc-41",
       "judge":{"model":"zen5-mini","criteria":"the answer names the right steps","name":"correctness"}}'

Attempts on the benchmark plane are append-only and keyed by benchmark, item and model, so re-queuing a run you already made is close to free, and a benchmark id outside the catalog is a 422 naming it rather than a partial run. The eval run is synchronous: it answers items, scored, avgScore and one row per example, and a run where nothing scored is a 502 rather than a 200 with an empty table.

What does not carry

No rails language. Colang describes dialog as flows — what the bot may talk about, what it says when asked to go elsewhere, how a multi-turn exchange should proceed — and Guardrails executes them around every turn. Nothing here interprets a flow file. Input and output decisions are Guard's fixed stages, or a zen-guard call whose answer your code acts on; topical and conversational control is the agent's instructions; what it may do is its tools list. A Colang configuration doing substantive dialog control is rewritten as those three, not ported.

No data curation or anonymization service. There is no address that deduplicates a corpus or rewrites PII in a dataset. Curator, Anonymizer and Safe Synthesizer remain the tools for that work and run where they ran; the change is the provider block. Synthesis is gym synth, a binary rather than an address: it covers Data Designer's sampler, expression and LLM columns, validators and judge, and leaves out the persona samplers, code and SQL validators, and MCP tool columns. zen-guard classifies a prompt at request time and does not touch training data.

Hosted fine-tuning is a job; RL is a binary you run. POST /v1/ai/finetune/jobs submits a training job with a base model, a method and a dataset, and POST /v1/ai/finetune/deploy serves the result under a model name. That is Customizer's shape, and it has no rollout loop. GRPO is hanzoai/gym: rl: grpo in an Axolotl-shaped config, verifiers named under trl.reward_funcs (exact_match, numeric_match, format:, length:, http:), LoRA on Qwen3 or Llama, on your own CUDA or Metal accelerators. It is single-node and LoRA-only; NeMo RL's multi-node full-parameter runs stay in NeMo, and either way the output is an adapter or checkpoint that /v1/ai/finetune/deploy or Engine serves.

No NIM to pull. Inference here is served, not shipped as a container, so there is no image whose version you pin and no GPU you size for it. The open-source engine behind the Zen models, hanzoai/engine, runs the same /v1 surface on your own hardware if you need one, but it is a project and not an address on this API.

No red-teaming harness. Auditor is garak with NVIDIA's probe set: it generates attacks, runs them, and scores what got through. There is no capability that generates attacks. zen-guard is the classifier that sits on the other side of that test. Point garak's openai.OpenAICompatible generator at https://api.hanzo.ai/v1/ and it measures the same thing it measured before; point it at a guard-proxy in front of that host and the difference between the two runs is what Guard's stages caught.

Relay's runtime events have no bus. Relay emits scope lifecycle events — started, tool called, model called, ended — that plugins subscribe to in process. What exists here is the trace after the fact and the session stream during an agent run at GET /v1/agent/sessions/stream. A plugin that acted on a Relay event before the call proceeded has no hook to attach to; the decision moves into the agent definition or into your caller.