# ML URL: https://docs.hanzo.ai/docs/openapi/ml > Model serving: deploy a model behind an endpoint and call it. Also for this capability: **API** · [CLI](https://docs.hanzo.ai/docs/cli/ml) · [SDKs](https://docs.hanzo.ai/docs/sdks) Model serving: deploy a model behind an endpoint and call it. > [All API references →](https://docs.hanzo.ai/docs/openapi) · [Get started →](https://docs.hanzo.ai/docs/guides) | | | | -------------- | -------------------------------------- | | **Base URL** | `https://api.hanzo.ai` | | **Operations** | 7 | | **Auth** | `Authorization: Bearer $HANZO_API_KEY` | ## Specification > **HIP-1140 · ML — Model Serving** — Draft · [read the specification →](https://github.com/hanzoai/hips/blob/7ed737180339cd05b5f2935eafe7f7b83d2449a2/HIPs/hip-1140-ml-model-serving.md) `/v1/ml` is model serving: deploy a model behind an endpoint and call it. The implementation, `hanzoai/cloud` `apps/ml`, is a thin tenant-scoped bridge that turns one KServe CustomResource — `InferenceService`, `serving.kserve.io/v1beta1` — into a small REST surface plus one inference proxy. No ML logic is reimplemented: KServe owns reconciliation, and this layer owns exactly two things, the REST↔Kubernetes translation and the tenant boundary (`apps/ml/ml.go:1-16`). ### Motivation Serving a model is cluster work — a CR, a namespace, a data-plane address — and every customer who had to be handed `kubectl` to do it was handed the cluster. The capability exists so a tenant deploys and calls a model through the same identity, billing and document planes as every other `/v1` surface, while the cluster remains invisible. Training is deliberately NOT here: a `/v1/train/*` facade over CRDs the cluster never served was deleted rather than shipped degraded (`apps/ml/ml.go:18-25`); fine-tuning is the `hanzoai/ai` broker's. ### Specification The key words MUST, MUST NOT, SHOULD and MAY are to be interpreted as in RFC 2119\. #### The store, and there is none This capability owns no store. Its state is the cluster's: every read and write is a call on a Kubernetes dynamic client, and deleting the subsystem loses nothing but the door. The client authenticates as the dedicated `cloud-ml` ServiceAccount when `HANZO_ML_TOKEN_FILE` names a mounted token, failing closed if the named token is unreadable (`apps/ml/ml.go:659-681`) — the product-API path never inherits ML's cluster reach. #### Addresses Seven operations under `/v1/ml`, three typed and four declared with prose beside the route, each declared one for a wire fact a typed op cannot state (`apps/ml/ml.go:210-241`): - `GET /v1/ml/models`, `GET /v1/ml/models/{name}`, `DELETE /v1/ml/models/{name}` — typed ops (`apps/ml/typed.go`). - `POST /v1/ml/models` — declared: the pre-create balance gate answers 402/503 in band with the fleet's nested `{"error":{code,message}}` body, which zip renders only flat. - `PATCH /v1/ml/models/{name}` — declared: an RFC 7386 merge patch relayed verbatim; re-encoding a merge patch changes what it means. - `POST /v1/ml/models/{name}/predict` — declared: a verbatim proxy of the predictor's own status, bytes and Content-Type (KServe v2 inference protocol), read up to a 32 MiB ceiling. - `GET /v1/ml/health` — declared: a real probe that answers 503 CARRYING the degraded report — k8s reachability, CRD presence, and the serving-runtime count, which is a separate fact because a cluster with the CRD and no runtime accepts a deploy and never schedules it (`apps/ml/ml.go:486-543`). #### Tenancy The tenant is the gateway-minted org narrowed by the validated project, mapped to a per-tenant Kubernetes namespace: `ml-`, or `ml--` for a non-default project. The mapping MUST be injective — org and project are validated against strict DNS-label patterns with no lossy fold (`apps/ml/ml.go:564-598`) — and the dynamic client is always pinned to the caller's namespace, so naming into another tenant's resources is unspellable rather than refused. An unvalidated principal is 403 before any mapping; an org-less caller is refused unless SuperAdmin, who lands in the literal `ml-admin` bucket. #### Money Metered (`plugin/ml/main.go` declares `cloud.Metered`; `ml` is in the `meteredApps` standing list, `spend.go:306`). One create is gated and debited through the shared `cloud.ResourceMeter` under the commerce product label `compute`, not `ml` (`apps/ml/ml.go:145-171`): the balance gate runs BEFORE the namespace or CR exists and fails closed, the fee (`CLOUD_COMPUTE_FEE_CENTS[_INFERENCESERVICE]`, default $1.00) is debited asynchronously on success against the caller's own ledger. Ongoing GPU-hour cost reuses the same meter from a runtime usage watcher and is never fabricated here. #### Events and telemetry It publishes nothing to the bus, so a customer's webhooks receive nothing from it. Beyond the request span every route gets, it emits log lines only — the mount report and per-operation Kubernetes errors with the missing RBAC named (`apps/ml/ml.go:636-644`). #### Stage `beta`: the manifest row (`manifest/apps.go:186`) declares `Stage: Beta`, so per HIP-0139 §8 the capability is dropped from the public projection and its prefix answers 404 unless the caller's org holds the `ml` flag. #### Upstream - KServe (Apache-2.0): the reconciler this capability drives. Nothing of it is vendored — the app speaks its CR API and its v2 inference wire. - `k8s.io/client-go` and `k8s.io/apimachinery` (Apache-2.0): the imported client through which every cluster call goes. ### Rationale The alternative to a thin bridge is reimplementing serving — a scheduler, a rollout loop, a data plane. That buys independence from KServe and costs a second reconciler that must agree with the cluster about everything. The bridge keeps one owner per fact: KServe owns the model's lifecycle, this layer owns who may name it. Verbatim relays (patch, predict) are kept verbatim for the same reason — a paraphrased predictor error is this layer inventing an answer the model did not give. ### Security Considerations The wrong implementation hands out the cluster. Concretely: a lossy org sanitizer folds two orgs onto one namespace and each reads the other's models and predictions; a namespace derived from a request field instead of the validated principal lets any caller deploy into, or predict against, a chosen tenant; and a client built on the pod's own broad ServiceAccount instead of the `cloud-ml` token gives every ML bug the product API's full cluster reach. The strict-regex injective mapping, the principal-only tenant resolution and the dedicated token are each the closure of one of those, and the balance gate running before resource creation is what keeps an unfunded org from starting GPU work it will never pay for. ## Four surfaces | Surface | Reaches this capability as | Coverage | | -------- | ------------------------------------------ | ------------------------------------------------------------------------- | | **REST** | `ml` at its own prefix | 7 operations | | **CLI** | `hanzo ml …` | 7 of 7 | | **SDK** | `MlApi` in every published client | 7 methods | | **MCP** | tool `ml` on `https://api.hanzo.ai/v1/mcp` | 4 operations, 0 under the document's own id — ask `describe` for the rest | ## Quickstart ```bash export HANZO_API_KEY=sk-... # console.hanzo.ai → API keys ``` Then the first call — a read that needs nothing but the key. `GET /v1/ml/health`, operation `get_ml_health`: **CLI** ```bash hanzo ml health ``` **SDK** **.ts** ```ts import { Configuration, MlApi } from 'hanzoai'; const api = new MlApi(new Configuration({ accessToken: process.env.HANZO_API_KEY })); const { data } = await api.getMlHealth(); ``` **.py** ```python from hanzoai.cloud import ApiClient, Configuration from hanzoai.cloud.api import MlApi client = ApiClient(Configuration(access_token=os.environ["HANZO_API_KEY"])) result = MlApi(client).get_ml_health() ``` **.go** ```go cfg := hanzoai.NewConfiguration() cfg.AddDefaultHeader("Authorization", "Bearer "+os.Getenv("HANZO_API_KEY")) client := hanzoai.NewAPIClient(cfg) resp, _, err := client.MlAPI.GetMlHealth(context.Background()).Execute() if err != nil { return err } ``` **.rs** ```rust use hanzo_client::apis::{configuration::Configuration, ml_api}; let mut cfg = Configuration::new(); cfg.bearer_access_token = std::env::var("HANZO_API_KEY").ok(); let result = ml_api::get_ml_health(&cfg, Default::default()).await?; ``` **.java** ```java import ai.hanzo.cloud.ApiClient; import ai.hanzo.cloud.api.MlApi; ApiClient client = new ApiClient(); client.setBearerToken(System.getenv("HANZO_API_KEY")); var result = new MlApi(client).getMlHealth(); ``` **HTTP** ```bash curl https://api.hanzo.ai/v1/ml/health \ -H "Authorization: Bearer $HANZO_API_KEY" ``` **MCP** MCP reaches **ml** through the `ml` tool, which names its 4 operations with its own verbs — this one among them, under a name only MCP declares. `describe` explains any of them: ```bash curl -X POST https://api.hanzo.ai/v1/mcp \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": { "name": "describe", "arguments": { "op": "list_ml_models" } } }' ``` ## Endpoints | Endpoint | What it does | | ----------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | [`GET /v1/ml/health`](https://docs.hanzo.ai/docs/openapi/ml/get-ml-health) | Whether model serving can actually work right now | | [`POST /v1/ml/models/{name}/predict`](https://docs.hanzo.ai/docs/openapi/ml/post-ml-models-by-name-predict) | Run inference against one of your deployed models | | [`GET /v1/ml/models/{name}`](https://docs.hanzo.ai/docs/openapi/ml/get-ml-models-by-name) | Returns one deployed inference model. | | [`PATCH /v1/ml/models/{name}`](https://docs.hanzo.ai/docs/openapi/ml/patch-ml-models-by-name) | Change a deployed model in place | | [`DELETE /v1/ml/models/{name}`](https://docs.hanzo.ai/docs/openapi/ml/delete-ml-models-by-name) | Deletes a deployed inference model. | | [`GET /v1/ml/models`](https://docs.hanzo.ai/docs/openapi/ml/get-ml-models) | Lists the inference models deployed in the caller's org. | | [`POST /v1/ml/models`](https://docs.hanzo.ai/docs/openapi/ml/post-ml-models) | Deploys one inference model for the caller's org, and answers 201 with the model as Kubernetes admitted it. | --- [All Hanzo APIs](https://docs.hanzo.ai/docs/openapi) · [Interactive reference](https://docs.hanzo.ai/docs/openapi/reference)