O11y

Package o11y is your logs, metrics and traces: ship them in, query them, chart them.

o11y

POST /v1/o11y/alerts/{receiver}

Take an Alertmanager notification and page a human

Records one Alertmanager webhook delivery and pages the on-call. Each alert prints an ALERT-RECEIVED line and joins the replay ring, then the batch is carried out of the process by the egress chain: the org's KMS-custodied Slack bot token first (the ONE product Slack egress, not a second webhook credential), falling back to a plain POST to CLOUD_ALERTS_WEBHOOK_URL — which needs no Slack connection and so works in exactly the state that silences the first. Resolved notifications page too: "it recovered" is the half of an incident people are actually waiting for.

THE STATUS CODE REPORTS DELIVERY, NOT ARRIVAL. 200 ok means an egress accepted the batch. If none did — including when none is configured at all — it answers 503 naming the failure, so Alertmanager retries and counts it in alertmanager_notifications_failed_total. An alert nobody could be told about must never answer the same way as one that was delivered.

A body that will not parse is still recorded (with empty fields) rather than rejected: the delivery happened, which is the fact being recorded, and a 400 would make Alertmanager retry a malformed payload forever.

The receiver segment is Alertmanager's own receiver name, a parameter rather than a hand-listed route because the receiver set is config, not code.

GET /v1/o11y/alerts/last

Replay the alert records this process took

Answers the most recent Alertmanager deliveries THIS process received, as plain text — one greppable ALERT-RECEIVED line per alert, followed by the ALERT-DELIVERED / ALERT-UNDELIVERED outcome of carrying it out of the process, newest last, so piping to tail reads in arrival order. (none) when nothing has landed.

Arrival and delivery are separate lines because they are separate facts that fail independently. Alertmanager can tell you it dispatched a notification, never that anything received it; this process taking the call says nothing about whether a human was reached. Reading only the first as if it were the second is how a pager stays silent for months behind a log where everything looks fine.

The ring is PROCESS-LOCAL and bounded to the last 200 lines. Both are the point: a record that outlived the process that took the call would be a claim about something nobody observed, and an unbounded log is a memory leak with a nice name. A restart empties it.

GET /v1/o11y/alerts

Returns the org's current alerts. Viewer gate.

POST /v1/o11y/api/{project_id}/envelope/

Receive a Sentry envelope on the SDK's own DSN path

Accepts an application/x-sentry-envelope frame from a Sentry SDK — the batched wire format carrying events, sessions and attachments — and ingests it against the project named in the path.

THE /api/ SEGMENT IS NOT OURS TO NAME. An SDK appends its own fixed /api/<project>/envelope/ suffix to whatever DSN it is given, so this address is the SDK's, received verbatim. We receive this shape; we do not publish it. The clean spelling of the same wire is /v1/sentry/{project}/envelope/.

AUTHENTICATED BY THE DSN PUBLIC KEY, never a Hanzo session, and therefore exempt from the principal gate: the ingest verifier checks the key in constant time, fails closed, and derives the org from it. A keyless submission is a 401 from that verifier — not a 403 from the gate, and not a 404 — which is how you tell the hops apart. The exemption is matched by method plus prefix plus suffix, never a bare prefix, so no read is reachable through it.

POST /v1/o11y/api/{project_id}/store/

Receive a single Sentry event on the SDK's own DSN path

The legacy single-event form of the envelope ingest: one JSON event rather than a framed batch, kept because SDKs in the field still send it.

Same address ownership and same authentication as the envelope route — the /api/ segment is the SDK's, the DSN public key is the credential, the principal gate does not apply, and a keyless submission is a 401 from the ingest verifier.

POST /v1/o11y/authz/check

Evaluates a batch of transactions — relation plus object — for the authenticated caller and answers each with its authorization verdict, in the order they were asked. It is the read a UI uses to decide which controls to show.

POST /v1/o11y/auto_complete/attribute_values

Reads the attribute-value request from the body rather than off the query string — the spelling the newer builder uses to send its filters alongside the request.

Callers need the viewer role; the runtime's own gate enforces it.

Request body — application/json (required)

GET /v1/o11y/autocomplete/aggregate_attributes

Lists the attributes usable as an aggregate target for the given telemetry and operator — what a filter builder offers after the aggregation is chosen.

Callers need the viewer role; the runtime's own gate enforces it.

GET /v1/o11y/autocomplete/attribute_keys

Lists the attribute keys available for filtering the given telemetry, each with its data type and whether it is a materialized column.

Callers need the viewer role; the runtime's own gate enforces it.

GET /v1/o11y/autocomplete/attribute_values

Lists the values one attribute key has taken — string, number and bool values in their own lists — for completing a filter.

Callers need the viewer role; the runtime's own gate enforces it.

GET /v1/o11y/availability

Reports how much of the Hanzo fleet is up — the current per-service inventory plus an up-versus-reporting trend across the window. Both come from the fleet prober's own measurements: every service is asked its health URL every 30 seconds, so a service is listed as down because it did not answer, never because something failed to collect it. PLATFORM SUDO ONLY — this is the whole fleet's inventory, not tenant data, so every customer is 403. An unreachable telemetry store answers 503 rather than an empty trend, because a board of zeroes and a fleet that is down look identical.

GET /v1/o11y/channels/{id}

Returns one notification channel, by id. Viewer gate.

PUT /v1/o11y/channels/{id}

Replaces a notification channel's receiver, by id. Admin gate.

Request body — application/json (required)

DELETE /v1/o11y/channels/{id}

Removes a notification channel, by id. Admin gate.

POST /v1/o11y/channels/test

Sends a test notification to the posted receiver. Editor gate.

Request body — application/json (required)

GET /v1/o11y/channels

Lists the org's notification channels. Viewer gate.

POST /v1/o11y/channels

Creates a notification channel, answering with the stored channel. Admin gate.

Request body — application/json (required)

GET /v1/o11y/cloud_integrations/{cloud_provider}/accounts/{id}/services/{service_id}

Returns one service and its configuration for a connected account of the given provider, by account id and service id. Admin gate.

PUT /v1/o11y/cloud_integrations/{cloud_provider}/accounts/{id}/services/{service_id}

Changes a service's configuration for one connected account of the given provider, by account id and service id. Admin gate.

Request body — application/json (required)

GET /v1/o11y/cloud_integrations/{cloud_provider}/accounts/{id}/services

Lists the services metadata for one connected account of the given provider, by account id. Admin gate.

GET /v1/o11y/cloud_integrations/{cloud_provider}/accounts/{id}

Returns one connected account for the given provider, by id. Admin gate.

PUT /v1/o11y/cloud_integrations/{cloud_provider}/accounts/{id}

Changes a connected account's configuration for the given provider, by id. Admin gate.

Request body — application/json (required)

DELETE /v1/o11y/cloud_integrations/{cloud_provider}/accounts/{id}

Tears down a connected account for the given provider, by id. Admin gate.

POST /v1/o11y/cloud_integrations/{cloud_provider}/accounts/check_in

Is the deployed agent's check-in — the path consistent with the account surface, reporting the agent's account and telemetry state so the connection can be tracked. Viewer gate — the agent's role is viewer.

Request body — application/json (required)

GET /v1/o11y/cloud_integrations/{cloud_provider}/accounts

Lists the cloud-integration accounts connected for the given provider. Admin gate.

POST /v1/o11y/cloud_integrations/{cloud_provider}/accounts

Connects a new cloud-integration account for the given provider from its posted config and credentials, answering with the account and the artifact the agent deploys to complete the connection. Admin gate.

Request body — application/json (required)

GET /v1/o11y/cloud_integrations/{cloud_provider}/credentials

Returns the credentials the connecting agent needs to establish the cloud integration, for the given cloud provider. Admin gate.

GET /v1/o11y/cloud_integrations/{cloud_provider}/services/{service_id}

Returns one service the given provider can collect from, by service id, optionally scoped to one cloud integration. Admin gate.

GET /v1/o11y/cloud_integrations/{cloud_provider}/services

Lists the services the given provider can collect from, optionally scoped to one cloud integration. Admin gate.

POST /v1/o11y/cloud-integrations/{cloud_provider}/agent-check-in

Is the deployed agent's check-in on its original hyphenated path, kept for backward compatibility with agents already running. Viewer gate — the agent's role is viewer.

Request body — application/json (required)

GET /v1/o11y/clusters/attribute_keys

Lists the metric attribute keys Kubernetes clusters report, for building cluster filters.

GET /v1/o11y/clusters/attribute_values

Lists the values one cluster attribute key has taken, for building cluster filters.

POST /v1/o11y/clusters/list

Lists Kubernetes clusters over a time range, each with its CPU and memory usage against allocatable capacity and its attributes; filterable, groupable and paginated.

Request body — application/json (required)

GET /v1/o11y/complete/google

Complete a Google sign-in

The callback Google redirects a user back to after they approve the sign-in. It exchanges the authorization code, establishes the session and answers 303 to the console.

The answer is a Location header and no body, which is why it is not a typed operation — declaring a JSON response for a redirect would publish a shape that does not exist and hide the header that is the entire point.

UNAUTHENTICATED by necessity: it is how a caller GETS a principal, so requiring one would be circular. It is not an open door — the code it carries is single-use and verified against the provider.

GET /v1/o11y/complete/oidc

Complete a generic OIDC sign-in

The callback any configured OIDC provider redirects back to. Same shape and same reasoning as the Google callback: the code is exchanged, the session is established, and the answer is a 303 to the console rather than a body.

UNAUTHENTICATED by necessity — this is the act of obtaining a principal, and the provider's own code is what authenticates it.

POST /v1/o11y/complete/saml

Complete a SAML sign-in

The assertion consumer service: the identity provider POSTs its signed assertion here, and a valid one establishes the session and answers 303 to the console.

A redirect, not a value, so it is not a typed operation. UNAUTHENTICATED by necessity and authenticated in fact by the assertion's signature, which is checked against the configured provider before any session exists.

POST /v1/o11y/countErrors

Counts the grouped exceptions in the query window for the caller's org.

Callers need the viewer role; the runtime's own gate enforces it.

Request body — application/json (required)

GET /v1/o11y/daemonsets/attribute_keys

Lists the metric attribute keys Kubernetes daemonsets report, for building daemonset filters.

GET /v1/o11y/daemonsets/attribute_values

Lists the values one daemonset attribute key has taken, for building daemonset filters.

POST /v1/o11y/daemonsets/list

Lists Kubernetes daemonsets over a time range, each with the CPU and memory its pods used against request and limit, desired and available node counts, restarts and attributes; filterable, groupable and paginated.

Request body — application/json (required)

PUT /v1/o11y/dashboard_views/{id}

Replaces a saved view's name and data. Saved views are shared org-wide.

Callers need the editor role; the runtime's own gate enforces it.

Request body — application/json (required)

DELETE /v1/o11y/dashboard_views/{id}

Removes a saved view. Saved views are shared org-wide. Deleting a non-existent view refuses with the runtime's not-found.

Callers need the editor role; the runtime's own gate enforces it.

GET /v1/o11y/dashboard_views

Returns every saved view in the calling user's org. Saved views are shared org-wide.

Callers need the viewer role; the runtime's own gate enforces it.

POST /v1/o11y/dashboard_views

Persists the calling user's dashboard-listing state (query, sort, order) as a named, reusable view shared across the org.

Callers need the editor role; the runtime's own gate enforces it.

Request body — application/json (required)

POST /v1/o11y/dashboards/{id}/clone

Clones an existing v2-shape dashboard. User and integration dashboards can be cloned; system dashboards are rejected. The clone keeps the source's display name, panels and tags, but gets a freshly generated unique internal name and is always created as an unlocked user dashboard owned by the caller.

Callers need the editor role; the runtime's own gate enforces it.

PUT /v1/o11y/dashboards/{id}/lock

Locks a v2-shape dashboard. Only the dashboard's creator or an org admin may lock or unlock.

Callers need the editor role; the runtime's own gate enforces it.

DELETE /v1/o11y/dashboards/{id}/lock

Unlocks a v2-shape dashboard. Only the dashboard's creator or an org admin may lock or unlock.

Callers need the editor role; the runtime's own gate enforces it.

GET /v1/o11y/dashboards/{id}/public

Returns the public-sharing config for a dashboard.

Callers need the admin role; the runtime's own gate enforces it.

POST /v1/o11y/dashboards/{id}/public

Creates the public-sharing config for a dashboard and enables public sharing, answering with the new share's id.

Callers need the admin role; the runtime's own gate enforces it.

Request body — application/json (required)

PUT /v1/o11y/dashboards/{id}/public

Updates the public-sharing config for a dashboard.

Callers need the admin role; the runtime's own gate enforces it.

Request body — application/json (required)

DELETE /v1/o11y/dashboards/{id}/public

Deletes the public-sharing config and disables public sharing of a dashboard.

Callers need the admin role; the runtime's own gate enforces it.

GET /v1/o11y/dashboards/{id}

Returns a v2-shape dashboard.

Callers need the viewer role; the runtime's own gate enforces it.

PUT /v1/o11y/dashboards/{id}

Updates a v2-shape dashboard's metadata, spec and tag set. The name is immutable and locked dashboards are rejected.

Callers need the editor role; the runtime's own gate enforces it.

Request body — application/json (required)

PATCH /v1/o11y/dashboards/{id}

Applies an RFC 6902 JSON Patch to a v2-shape dashboard. The patch is applied against the postable view (metadata, spec, tags), so individual panels, queries, variables, layouts or tags can be updated without re-sending the rest. Apply is lenient — remove on a missing path is a no-op and add creates any missing parent objects — and the result is still validated. Locked dashboards are rejected. The request body is the bare JSON Patch operations array.

Callers need the editor role; the runtime's own gate enforces it.

Request body — application/json (required)

DELETE /v1/o11y/dashboards/{id}

Deletes a v2-shape dashboard along with its tag relations. Locked dashboards are rejected.

Callers need the editor role; the runtime's own gate enforces it.

GET /v1/o11y/dashboards

Returns a page of v2-shape dashboards for the org. This is the pure, user-independent list — it carries no pin state; use dashboardListForUserV2 for the personalized, pin-aware list. Supports a filter DSL (query), sort (updated_at/created_at/name), order (asc/desc), and offset-based pagination (limit/offset).

Callers need the viewer role; the runtime's own gate enforces it.

POST /v1/o11y/dashboards

Creates a dashboard in the v2 format that follows the Perses spec and answers with the stored dashboard.

Callers need the editor role; the runtime's own gate enforces it.

Request body — application/json (required)

POST /v1/o11y/dependency_graph

Returns the service dependency graph over the requested window: every parent→child edge observed, with call and error rates and latency percentiles per edge.

Callers need the viewer role; the runtime's own gate enforces it.

Request body — application/json (required)

GET /v1/o11y/deployments/attribute_keys

Lists the metric attribute keys Kubernetes deployments report, for building deployment filters.

GET /v1/o11y/deployments/attribute_values

Lists the values one deployment attribute key has taken, for building deployment filters.

POST /v1/o11y/deployments/list

Lists Kubernetes deployments over a time range, each with the CPU and memory its pods used against request and limit, desired and available replica counts, restarts and attributes; filterable, groupable and paginated.

Request body — application/json (required)

GET /v1/o11y/disks

Lists the storage disks the datastore reports, with their names and types.

Callers need the viewer role; the runtime's own gate enforces it.

GET /v1/o11y/domains/{id}

Returns one auth domain with its SSO configuration, by id. Admin gate.

PUT /v1/o11y/domains/{id}

Replaces one auth domain's SSO configuration, by id. Admin gate.

Request body — application/json (required)

DELETE /v1/o11y/domains/{id}

Releases an email domain and discards its SSO configuration, by id. Admin gate.

GET /v1/o11y/domains

Lists the org's auth domains — the email domains whose SSO configuration this org owns. Admin gate.

POST /v1/o11y/domains

Claims an email domain for the org and configures how its users sign in; the answer is the new domain's id. Admin gate.

Request body — application/json (required)

GET /v1/o11y/downtime_schedules/{id}

Returns one planned maintenance window, by id. Viewer gate.

PUT /v1/o11y/downtime_schedules/{id}

Replaces a planned maintenance window, by id. Editor gate.

Request body — application/json (required)

DELETE /v1/o11y/downtime_schedules/{id}

Removes a planned maintenance window, by id. Editor gate.

GET /v1/o11y/downtime_schedules

Lists all planned maintenance windows, optionally narrowed to the active ones or the recurring ones. Viewer gate.

POST /v1/o11y/downtime_schedules

Creates a planned maintenance window, answering with the stored schedule. Editor gate.

Request body — application/json (required)

GET /v1/o11y/errorFromErrorID

Returns one exception instance and the span it happened on, by its error id within a group at a timestamp.

Callers need the viewer role; the runtime's own gate enforces it.

GET /v1/o11y/errorFromGroupID

Returns the representative exception instance of a group at a timestamp, and the span it happened on.

Callers need the viewer role; the runtime's own gate enforces it.

GET /v1/o11y/errortracking/issues/{id}

Returns one grouped issue with its latest occurrence sample.

Callers need the viewer role; the runtime's own gate enforces it.

POST /v1/o11y/errortracking/issues/{id}

Changes an issue's lifecycle — resolve, ignore, reopen or assign — and returns the updated issue. Fields left unset are left unchanged.

Callers need the editor role; the runtime's own gate enforces it.

Request body — application/json (required)

GET /v1/o11y/errortracking/issues

Lists the caller's org's grouped error issues (by fingerprint) with status, level, counts and first/last-seen.

Callers need the viewer role; the runtime's own gate enforces it.

POST /v1/o11y/event

Records one product-analytics event for the signed-in user — a track event with a name and free-form attributes.

Callers need the viewer role; the runtime's own gate enforces it.

Request body — application/json (required)

GET /v1/o11y/explorer/views/{viewId}

Returns one saved explorer view by id.

Callers need the viewer role; the runtime's own gate enforces it.

PUT /v1/o11y/explorer/views/{viewId}

Replaces one saved explorer view by id with the given view and echoes it back.

Callers need the editor role; the runtime's own gate enforces it.

Request body — application/json (required)

DELETE /v1/o11y/explorer/views/{viewId}

Deletes one saved explorer view by id.

Callers need the editor role; the runtime's own gate enforces it.

GET /v1/o11y/explorer/views

Lists the caller's org's saved explorer views, optionally narrowed to one source page, name or category.

Callers need the viewer role; the runtime's own gate enforces it.

POST /v1/o11y/explorer/views

Saves a new explorer view for the caller's org and returns its id.

Callers need the editor role; the runtime's own gate enforces it.

Request body — application/json (required)

POST /v1/o11y/export_raw_data

Export raw telemetry rows as a file

Runs a query and returns its rows as a downloadable CSV or JSONL attachment, chunked, with a trailer that says whether the export completed — so a truncated download is detectable rather than silently short.

The answer is a file, not a value, which is why it is not a typed operation: the body is neither JSON nor bounded. Use the query operations when you want rows in a response.

A validated, org-scoped principal is required and the export carries that principal's own tenant only.

POST /v1/o11y/factor_password/forgot

Starts the forgotten-password flow: the named user is mailed a reset link. Unauthenticated by design, and deliberately quiet about whether the address exists.

Request body — application/json (required)

GET /v1/o11y/features

Returns the supported feature flags and their resolved values for the caller's org.

Callers need the viewer role; the runtime's own gate enforces it.

GET /v1/o11y/fields/keys

Returns the telemetry field keys matching the selector — the signal's fields grouped by name, and whether the catalog is complete.

Callers need the viewer role; the runtime's own gate enforces it.

GET /v1/o11y/fields/values

Returns the values one telemetry field has taken — string, bool, number and related values — and whether the value list is complete.

Callers need the viewer role; the runtime's own gate enforces it.

GET /v1/o11y/filter_suggestions

Suggests attribute keys and example filter queries for the query builder, seeded by what the org's own telemetry carries. Only the logs data source is supported today.

Callers need the viewer role; the runtime's own gate enforces it.

POST /v1/o11y/gateway/ingestion_keys/{keyId}/limits

Sets a signal limit on an ingestion key, by key id, answering with the created limit. Editor gate.

Request body — application/json (required)

PATCH /v1/o11y/gateway/ingestion_keys/{keyId}

Changes an ingestion key, by id. Editor gate.

Request body — application/json (required)

DELETE /v1/o11y/gateway/ingestion_keys/{keyId}

Removes an ingestion key, by id. Editor gate.

PATCH /v1/o11y/gateway/ingestion_keys/limits/{limitId}

Changes an ingestion key limit, by limit id. Editor gate.

Request body — application/json (required)

DELETE /v1/o11y/gateway/ingestion_keys/limits/{limitId}

Removes an ingestion key limit, by limit id. Editor gate.

GET /v1/o11y/gateway/ingestion_keys/search

Lists the workspace's ingestion keys whose name matches the search, paginated. Editor gate.

GET /v1/o11y/gateway/ingestion_keys

Lists the workspace's ingestion keys, paginated. Editor gate.

POST /v1/o11y/gateway/ingestion_keys

Mints an ingestion key for the workspace, answering with the created key. Editor gate.

Request body — application/json (required)

GET /v1/o11y/getResetPasswordToken/{id}

Returns a user's password-reset token, creating one if none is live. Deprecated in favor of the reset_password_tokens pair, which separates reading from minting. Admin gate.

GET /v1/o11y/global/config

Returns the deployment's global configuration: its public endpoints and which identity providers are enabled. Open by design — the sign-in page reads it before anyone is signed in.

Open by design; the runtime's own gate is OpenAccess.

GET /v1/o11y/health

Reports service health. With live set, the datastore connection is checked too and an unhealthy store refuses with 503.

Open by design; the runtime's own gate is OpenAccess.

GET /v1/o11y/healthz

Health of the observability runtime's services

Reports whether every service in the runtime's registry is healthy, and names them grouped by state — so a failure says WHICH component is down, not merely that something is. An unhealthy registry answers 503, not a 200 with a false flag inside, so a plain status check cannot read a sick runtime as well.

UNAUTHENTICATED by design, like the other two probes: it carries no tenant data and is reached by k8s and by external checks that hold no principal.

GET /v1/o11y/hosts/attribute_keys

Lists the metric attribute keys hosts report, for building host filters — each with its data type and whether it is a materialized column.

GET /v1/o11y/hosts/attribute_values

Lists the values one host attribute key has taken, for building host filters — string, number and bool values in their own lists.

POST /v1/o11y/hosts/list

Lists monitored hosts over a time range, each with its CPU, memory, I/O wait and 15-minute load, whether it is actively reporting, its OS and its attributes; filterable, groupable and paginated. The answer also says whether any host metrics were received at all and which clusters and nodes sent them, so an empty page is distinguishable from a fleet that never reported.

Request body — application/json (required)

GET /v1/o11y/infra_monitoring/checks

Reports whether the metrics and attributes an infra-monitoring section needs are being received — for each collector receiver or processor involved, what is present and what is missing, with a user-facing message and a docs link per missing piece. Ready is true only when nothing is missing.

POST /v1/o11y/infra_monitoring/clusters

Lists Kubernetes clusters with CPU and memory usage against allocatable capacity summed over their nodes, plus per-group node readiness and pod phase counts. Rows answer as 'list' under the default k8s.cluster.name grouping or 'grouped_list' under a custom groupBy; a metric with no data in the window answers -1. Filterable by expression, orderable by usage or allocatable, paginated by offset and limit.

Request body — application/json (required)

POST /v1/o11y/infra_monitoring/daemonsets

Lists Kubernetes daemonsets with the CPU and memory their pods used against request and limit, the latest desired and current scheduled NODE counts (node counts, not pod counts), and per-group pod phase counts. Rows answer as 'list' under the default k8s.daemonset.name grouping or 'grouped_list' under a custom groupBy; a metric with no data in the window answers -1. Filterable by expression, orderable by the pod metrics or the node counts, paginated by offset and limit.

Request body — application/json (required)

POST /v1/o11y/infra_monitoring/deployments

Lists Kubernetes deployments with the CPU and memory their pods used against request and limit, the latest desired and available replica counts, and per-group pod phase counts. Rows answer as 'list' under the default k8s.deployment.name grouping or 'grouped_list' under a custom groupBy; a metric with no data in the window answers -1. Filterable by expression, orderable by the pod metrics or the replica counts, paginated by offset and limit.

Request body — application/json (required)

POST /v1/o11y/infra_monitoring/hosts

Lists hosts with key infrastructure metrics — CPU, memory, I/O wait and disk usage percentages and 15-minute load — plus an active/inactive status from whether the host reported in the last ten minutes. Rows answer as 'list' under the default host.name grouping or 'grouped_list' under a custom groupBy; a metric with no data in the window answers -1. Filterable by expression and by status, orderable by any of the five metrics, paginated by offset and limit.

Request body — application/json (required)

POST /v1/o11y/infra_monitoring/jobs

Lists Kubernetes jobs with the CPU and memory their pods used against request and limit, the latest desired-successful, active, failed and successful pod counters, and per-group pod phase counts — the phase counts are current state while the counters are cumulative over the job's life. Rows answer as 'list' under the default k8s.job.name grouping or 'grouped_list' under a custom groupBy; a metric with no data in the window answers -1. Filterable by expression, orderable by the pod metrics or the job counters, paginated by offset and limit.

Request body — application/json (required)

POST /v1/o11y/infra_monitoring/namespaces

Lists Kubernetes namespaces with the CPU and memory their pods used and per-group pod phase counts. Rows answer as 'list' under the default k8s.namespace.name grouping or 'grouped_list' under a custom groupBy, aggregating pods either way; a metric with no data in the window answers -1. Filterable by expression, orderable by cpu or memory, paginated by offset and limit.

Request body — application/json (required)

POST /v1/o11y/infra_monitoring/nodes

Lists Kubernetes nodes with CPU and memory usage against allocatable capacity, per-group readiness counts and per-group phase counts for the pods scheduled on them. Rows answer as 'list' under the default k8s.node.name grouping (each row one node with its readiness condition) or 'grouped_list' under a custom groupBy; a metric with no data in the window answers -1. Filterable by expression, orderable by usage or allocatable, paginated by offset and limit.

Request body — application/json (required)

POST /v1/o11y/infra_monitoring/pods

Lists Kubernetes pods with CPU and memory usage against request and limit, the pod's phase and its age, plus its namespace, node, owning workload and cluster attributes. Rows answer as 'list' under the default k8s.pod.uid grouping (each row one pod) or 'grouped_list' under a custom groupBy (each row aggregating its pods with per-phase counts); a metric with no data in the window answers -1. Filterable by expression, orderable by the six pod metrics, paginated by offset and limit.

Request body — application/json (required)

POST /v1/o11y/infra_monitoring/pvcs

Lists Kubernetes persistent volume claims with available, capacity and used bytes and inode counts, plus the claim's pod, namespace, node, statefulset and cluster attributes. Rows answer as 'list' under the default k8s.persistentvolumeclaim.name grouping or 'grouped_list' under a custom groupBy; a metric with no data in the window answers -1. Filterable by expression, orderable by the six volume metrics, paginated by offset and limit.

Request body — application/json (required)

POST /v1/o11y/infra_monitoring/statefulsets

Lists Kubernetes statefulsets with the CPU and memory their pods used against request and limit, the latest desired and current replica counts, and per-group pod phase counts. Rows answer as 'list' under the default k8s.statefulset.name grouping or 'grouped_list' under a custom groupBy; a metric with no data in the window answers -1. Filterable by expression, orderable by the pod metrics or the replica counts, paginated by offset and limit.

Request body — application/json (required)

GET /v1/o11y/infra_onboarding/k8s/status

Reports how far Kubernetes infra onboarding has progressed: which metric families have arrived and, per pod, which required metadata labels are present.

Callers need the viewer role; the runtime's own gate enforces it.

GET /v1/o11y/integrations/{integrationId}/connection_status

Reports whether the integration's logs and metrics have been received over the lookback window, so the console can show a live connection state. An integration that is not installed answers with an empty status rather than an error. Viewer gate.

GET /v1/o11y/integrations/{integrationId}

Returns one integration's full detail — its overview, configuration steps, collected data and assets — together with its installation record when the org has installed it. Viewer gate.

POST /v1/o11y/integrations/install

Installs an integration into the caller's org from its id and configuration, answering with the installed catalog item. Viewer gate.

Request body — application/json (required)

POST /v1/o11y/integrations/uninstall

Removes an integration from the caller's org by id. Viewer gate.

Request body — application/json (required)

GET /v1/o11y/integrations

Lists the available integrations and whether each is installed in the caller's org, optionally narrowed to installed or not-installed. Viewer gate.

POST /v1/o11y/invite/bulk

Invites several people to the caller's org in one call, refusing the whole batch when any email repeats. Deprecated alongside createInvite. Admin gate.

Request body — application/json (required)

POST /v1/o11y/invite

Invites one person to the caller's org by email, with the role they will hold when they accept. Deprecated in favor of creating users directly; kept because callers still hold it. Admin gate, enforced by the runtime this op relays to.

Request body — application/json (required)

GET /v1/o11y/jobs/attribute_keys

Lists the metric attribute keys Kubernetes jobs report, for building job filters.

GET /v1/o11y/jobs/attribute_values

Lists the values one job attribute key has taken, for building job filters.

POST /v1/o11y/jobs/list

Lists Kubernetes jobs over a time range, each with the CPU and memory its pods used against request and limit, desired-successful, active, failed and successful pod counts, restarts and attributes; filterable, groupable and paginated.

Request body — application/json (required)

GET /v1/o11y/licenses/active

Activates the enterprise license. This build has no enterprise edition, so the licensing provider refuses it as unsupported.

Callers need the viewer role; the runtime's own gate enforces it.

GET /v1/o11y/licenses

Lists the org's licenses. This build has no enterprise edition, so the list is intentionally empty.

Callers need the viewer role; the runtime's own gate enforces it.

POST /v1/o11y/listErrors

Lists the grouped exceptions in the query window — each an exception type with its message, count, service and first/last-seen — for the caller's org.

Callers need the viewer role; the runtime's own gate enforces it.

Request body — application/json (required)

GET /v1/o11y/livez

Liveness of the observability process

Answers 200 unconditionally while the process is running, and asserts NOTHING about the telemetry stores behind it. That is what makes it a liveness probe: a container that answers this is worth leaving alive, and restarting on a store outage would only remove the thing reporting the outage.

UNAUTHENTICATED by design, and one of exactly three /v1/o11y paths that are. It carries no tenant data, and gating it would break the k8s probes and the external health checks without protecting anything. Use the health probe, not this one, to ask whether the runtime can actually serve.

GET /v1/o11y/llm_pricing_rules/{id}

Returns a single LLM pricing rule by id.

Callers need the viewer role; the runtime's own gate enforces it.

DELETE /v1/o11y/llm_pricing_rules/{id}

Hard-deletes a pricing rule by id. If the rule was auto-synced, the next sync cycle recreates it.

Callers need the admin role; the runtime's own gate enforces it.

GET /v1/o11y/llm_pricing_rules

Returns the LLM pricing rules for the caller's org, with pagination and an optional search and override filter.

Callers need the viewer role; the runtime's own gate enforces it.

PUT /v1/o11y/llm_pricing_rules

Writes the pricing-rule batch — the single write endpoint used by both the user and the Zeus sync job. Per-rule match is by id, then sourceId, then insert; an override row is fully preserved when the request omits isOverride, only its synced_at stamped.

Callers need the admin role; the runtime's own gate enforces it.

Request body — application/json (required)

GET /v1/o11y/llm/annotation

Lists human annotations on traces and observations, optionally scoped to one review queue.

Callers need the viewer role; the runtime's own gate enforces it.

POST /v1/o11y/llm/annotation

Adds a human annotation to a trace or observation, optionally in a review queue.

Callers need the editor role; the runtime's own gate enforces it, and it validates the payload and stamps the annotation's author and org.

Request body — application/json (required)

GET /v1/o11y/llm/observations

Lists gen_ai spans as LLM observations — each an LLM call with its model, token counts, cost and latency projected from gen_ai.* attributes, newest first, over the query window.

Callers need the viewer role; the runtime's own gate enforces it, and scopes the read to the caller's validated tenant.

GET /v1/o11y/llm/score/{id}

Returns a single score by id.

Callers need the viewer role; the runtime's own gate enforces it.

DELETE /v1/o11y/llm/score/{id}

Hard-deletes a score by id.

Callers need the editor role; the runtime's own gate enforces it.

GET /v1/o11y/llm/scores

Lists eval scores and human-feedback signals attached to traces and observations, newest first.

Callers need the viewer role; the runtime's own gate enforces it.

POST /v1/o11y/llm/scores

Attaches an eval score or human-feedback signal to a trace or a single observation.

Callers need the editor role; the runtime's own gate enforces it, and it validates the payload and stamps the score's author and org.

Request body — application/json (required)

GET /v1/o11y/llm/sessions

Lists conversations — gen_ai spans grouped by session.id, with their trace and observation counts, tokens and cost.

Callers need the viewer role; the runtime's own gate enforces it.

GET /v1/o11y/llm/traces

Lists LLM traces — gen_ai spans grouped by trace_id, with cost, tokens and latency rolled up across each trace.

Callers need the viewer role; the runtime's own gate enforces it.

GET /v1/o11y/llm/users

Lists end users — gen_ai spans grouped by user.id, with their session, trace and observation counts, tokens and cost.

Callers need the viewer role; the runtime's own gate enforces it.

GET /v1/o11y/logs/aggregate

Returns the logs aggregate buckets for the query window. The runtime currently answers the empty set; the shape is the contract.

Callers need the viewer role; the runtime's own gate enforces it.

GET /v1/o11y/logs/fields

Returns the log field catalog: the fields already selected as indexed columns, and the interesting ones seen in the data that could be.

Callers need the viewer role; the runtime's own gate enforces it.

POST /v1/o11y/logs/fields

Changes how one log field is stored — selects or deselects it as a materialized column and tunes its index — and echoes the setting back.

Callers need the editor role; the runtime's own gate enforces it.

Request body — application/json (required)

GET /v1/o11y/logs/livetail

Follow log records as they arrive

Streams matching log records continuously instead of answering once, so a console tail shows lines as they land rather than at the end of a window.

It is a STREAM, which is why it is not a typed operation: there is no single complete value to name, and a generated client that waited for one would hang on the first tail. Read the bounded window with the log read instead when you want an answer rather than a feed.

A validated, org-scoped principal is required and the feed carries that principal's own tenant only.

GET /v1/o11y/logs/pipelines/{version}

Returns the caller's org's log parsing pipelines at one config version — "latest" for the newest — along with that version's deployment record and the recent version history.

Callers need the viewer role; the runtime's own gate enforces it.

POST /v1/o11y/logs/pipelines/preview

Runs the given log parsing pipelines over the given sample records without saving anything, and returns the transformed records plus whatever the collector logged while simulating them.

Callers need the viewer role; the runtime's own gate enforces it.

Request body — application/json (required)

POST /v1/o11y/logs/pipelines

Saves the given log parsing pipelines as the new config version for the caller's org and starts deploying it. The set REPLACES the current one: a pipeline left out of the request is dropped from the new version, and an empty set drops them all.

Callers need the editor role; the runtime's own gate enforces it.

Request body — application/json (required)

GET /v1/o11y/logs/promote_paths

Lists the log body paths already promoted or indexed, with the indexes each carries.

Callers need the viewer role; the runtime's own gate enforces it.

POST /v1/o11y/logs/promote_paths

Promotes and indexes log body paths: each named path is lifted out of the JSON body into its own column, with the indexes the caller asked for. Paths must start with "body.".

Callers need the editor role; the runtime's own gate enforces it.

GET /v1/o11y/logs

Returns the most recent log records in the query window, newest first — each record an open object carrying its nanosecond timestamp and whatever fields the record was ingested with.

Callers need the viewer role; the runtime's own gate enforces it.

POST /v1/o11y/messaging-queues/kafka/consumer-lag/consumer-details

Returns the consumer side of a consumer-lag view: the consumer groups reading the topic/partition named in variables, with their throughput and latency over the window.

Request body — application/json (required)

POST /v1/o11y/messaging-queues/kafka/consumer-lag/network-latency

Returns consumer network latency correlated per client: a throughput pass over the window finds the consumer clients, then their fetch latency joins in as a latency column per client/instance/service.

Request body — application/json (required)

POST /v1/o11y/messaging-queues/kafka/consumer-lag/producer-details

Returns the producer side of a consumer-lag view: the producers writing to the topic/partition named in variables, with their throughput and latency over the window.

Request body — application/json (required)

POST /v1/o11y/messaging-queues/kafka/onboarding/consumers

Checks whether the spans the Kafka consumer views need are arriving, row for row like producersOnboarding.

Request body — application/json (required)

POST /v1/o11y/messaging-queues/kafka/onboarding/kafka

Checks whether Kafka's own metrics — consumer lag and partition telemetry — are arriving, so the lag views can be lit up.

Request body — application/json (required)

POST /v1/o11y/messaging-queues/kafka/onboarding/producers

Checks whether the spans the Kafka producer views need are arriving — one row per required span attribute, with a pass/fail status and, on failure, what is missing from the instrumentation.

Request body — application/json (required)

POST /v1/o11y/messaging-queues/kafka/partition-latency/consumer

Returns the consumer-group latency detail for the topic and partition named in the request's variables.

Request body — application/json (required)

POST /v1/o11y/messaging-queues/kafka/partition-latency/overview

Returns the per-partition latency overview for the window — each topic/partition with its throughput and latency profile.

Request body — application/json (required)

POST /v1/o11y/messaging-queues/kafka/span/evaluation

Correlates producer and consumer spans over the evaluation window (eval_time bounds the scan) and returns the pairings with their end-to-end delay — the check that messages produced are being consumed.

Request body — application/json (required)

POST /v1/o11y/messaging-queues/kafka/topic-throughput/consumer-details

Breaks one consumer topic's throughput down using the topic and service named in variables.

Request body — application/json (required)

POST /v1/o11y/messaging-queues/kafka/topic-throughput/consumer

Returns the consumer topic-throughput overview for the window — what each consumer group read, per topic.

Request body — application/json (required)

POST /v1/o11y/messaging-queues/kafka/topic-throughput/producer-details

Breaks one producer topic's throughput down using the topic and service named in variables.

Request body — application/json (required)

POST /v1/o11y/messaging-queues/kafka/topic-throughput/producer

Returns the producer topic-throughput overview for the window — what each producer service wrote, per topic.

Request body — application/json (required)

POST /v1/o11y/messaging-queues/queue-overview

Lists the messaging destinations observed in the window — one row per queue/destination/service combination with its throughput and latency columns. Filters narrow by queue system, destination, service or any span attribute.

Request body — application/json (required)

GET /v1/o11y/metric_reduction_rules/{id}

Returns one volume-control rule by its id.

PUT /v1/o11y/metric_reduction_rules/{id}

Updates the match type and labels of a volume-control rule by its id; the metric name is immutable.

Request body — application/json (required)

DELETE /v1/o11y/metric_reduction_rules/{id}

Deletes a volume-control rule by its id.

POST /v1/o11y/metric_reduction_rules/preview

Estimates the series reduction and the dashboards and alerts a candidate volume-control rule would touch, without persisting it.

Request body — application/json (required)

GET /v1/o11y/metric_reduction_rules/stats

Returns total ingested vs retained series and samples and the estimated monthly savings across all volume-control rules.

GET /v1/o11y/metric_reduction_rules/timeseries

Returns ingested vs retained series over time across all volume-control rules, in hourly buckets, in the query-range time-series response shape.

GET /v1/o11y/metric_reduction_rules

Lists the org's metric volume-control (label reduction) rules, pageable and sortable by name, volume or recency.

POST /v1/o11y/metric_reduction_rules

Creates a volume-control rule for a metric and returns it with its id; a metric that already has a rule is refused.

Request body — application/json (required)

GET /v1/o11y/metric/metric_metadata

Serves the OLDER /metric/metric_metadata route. It is NOT the same op as metrics.go's metricMetadata (/metrics/metadata): different path, different input (this one also scopes by service). Two slices named one Go function for two routes; the route is the identity, so the name follows it. Renamed rather than merged — collapsing them would silently drop the service scope this one accepts. It returns one metric's metadata — its type, unit, description, temporality, monotonicity and histogram buckets — optionally scoped to the metric as one service reports it.

Callers need the viewer role; the runtime's own gate enforces it.

GET /v1/o11y/metrics/alerts

Lists the alert rules that reference a metric.

GET /v1/o11y/metrics/attributes

Returns one metric's attribute keys, each with its unique values and their count.

GET /v1/o11y/metrics/dashboards

Lists the dashboard panels that reference a metric.

GET /v1/o11y/metrics/highlights

Returns one metric's headline numbers: data points, total and active time series, and when it was last received.

POST /v1/o11y/metrics/inspect

Returns one metric's raw time series over a window of at most thirty minutes — each series with its labels and timestamp/value pairs.

Request body — application/json (required)

GET /v1/o11y/metrics/metadata

Returns one metric's metadata: description, type, unit, temporality and monotonicity.

POST /v1/o11y/metrics/metadata

Updates one metric's metadata — description, type, unit, temporality, monotonicity — and answers with the bare success envelope.

Request body — application/json (required)

GET /v1/o11y/metrics/onboarding

Reports whether any non-O11y metrics have been ingested — the lightweight check onboarding polls.

POST /v1/o11y/metrics/stats

Lists metrics with their sample and time-series counts for a time range — the volume view of the metrics explorer, pageable and sortable.

Request body — application/json (required)

POST /v1/o11y/metrics/treemap

Returns the proportional distribution of metrics by sample count or time-series count, as the entries of a treemap.

Request body — application/json (required)

GET /v1/o11y/metrics

Lists the distinct metric names seen in a time range, each with its description, type, unit, temporality and monotonicity.

GET /v1/o11y/namespaces/attribute_keys

Lists the metric attribute keys Kubernetes namespaces report, for building namespace filters.

GET /v1/o11y/namespaces/attribute_values

Lists the values one namespace attribute key has taken, for building namespace filters.

POST /v1/o11y/namespaces/list

Lists Kubernetes namespaces over a time range, each with the CPU and memory its pods used, their phase counts and its attributes; filterable, groupable and paginated.

Request body — application/json (required)

GET /v1/o11y/nextPrevErrorIDs

Returns the ids of the exception instances immediately after and before a given one within its group — the paging cursor the error detail view walks.

Callers need the viewer role; the runtime's own gate enforces it.

GET /v1/o11y/nodes/attribute_keys

Lists the metric attribute keys Kubernetes nodes report, for building node filters.

GET /v1/o11y/nodes/attribute_values

Lists the values one node attribute key has taken, for building node filters.

POST /v1/o11y/nodes/list

Lists Kubernetes nodes over a time range, each with its CPU and memory usage against allocatable capacity, readiness condition counts and attributes; filterable, groupable and paginated.

Request body — application/json (required)

GET /v1/o11y/org/preferences/{name}

Returns one org-scoped preference, by name. Admin gate.

PUT /v1/o11y/org/preferences/{name}

Sets one org-scoped preference, by name. Admin gate.

Request body — application/json (required)

GET /v1/o11y/org/preferences

Lists every org-scoped preference, each with its current and default value. Admin gate.

GET /v1/o11y/orgs/me/filters/{signal}

Returns the org's quick filters for one signal — traces, logs, metrics, exceptions or api_monitoring. Viewer gate.

GET /v1/o11y/orgs/me/filters

Returns the org's quick filters for every signal — the attribute shortlists its explorers offer as one-click filters. Viewer gate.

PUT /v1/o11y/orgs/me/filters

Replaces the org's quick filters for one signal with the attribute list given. Admin gate.

Request body — application/json (required)

GET /v1/o11y/orgs/me

Returns the caller's own organization. Admin gate.

PUT /v1/o11y/orgs/me

Rewrites the caller's own organization record — display name, name, alias — always addressed as "me", never by id. Admin gate.

Request body — application/json (required)

GET /v1/o11y/pods/attribute_keys

Lists the metric attribute keys Kubernetes pods report, for building pod filters.

GET /v1/o11y/pods/attribute_values

Lists the values one pod attribute key has taken, for building pod filters.

POST /v1/o11y/pods/list

Lists Kubernetes pods over a time range, each with its CPU and memory usage against request and limit, restart count, phase counts and attributes; filterable, groupable and paginated.

Request body — application/json (required)

GET /v1/o11y/processes/attribute_keys

Lists the metric attribute keys processes report, for building process filters.

GET /v1/o11y/processes/attribute_values

Lists the values one process attribute key has taken, for building process filters.

POST /v1/o11y/processes/list

Lists monitored processes over a time range, each with its name, PID, command line and CPU and memory usage; filterable, groupable and paginated.

Request body — application/json (required)

GET /v1/o11y/product/metrics

Returns one product's RED series — request rate, errors, p50 and p95 latency — for the caller's org, plus that org's LLM usage rollup over the same window. The series come from org-tagged request spans, so a tenant only ever aggregates its own traffic; a validated platform SuperAdmin sees the whole product's RED, while usage stays the caller's own org either way. A well-formed product with no backing workload answers empty series; a malformed slug is a 400.

GET /v1/o11y/public/dashboards/{id}/widgets/{idx}/query_range

Returns the query-range result for one widget of a public dashboard. When the share fixes its own time range the caller's startTime/endTime are ignored; otherwise they bound the window as millisecond epochs.

Anonymous, scoped to the public dashboard's read scope; the runtime's own gate enforces it.

GET /v1/o11y/public/dashboards/{id}

Returns the sanitized dashboard data for public access — the read a shared dashboard's public page makes.

Anonymous, scoped to the public dashboard's read scope; the runtime's own gate enforces it.

GET /v1/o11y/pvcs/attribute_keys

Lists the metric attribute keys persistent volume claims report, for building volume filters.

GET /v1/o11y/pvcs/attribute_values

Lists the values one persistent-volume-claim attribute key has taken, for building volume filters.

POST /v1/o11y/pvcs/list

Lists Kubernetes persistent volume claims over a time range, each with its available, capacity and used bytes, inode counts and attributes; filterable, groupable and paginated.

Request body — application/json (required)

POST /v1/o11y/query_filter/analyze

Analyzes a query and extracts the metric names it reads and the columns it groups by.

Callers need the viewer role; the runtime's own gate enforces it.

Request body — application/json (required)

GET /v1/o11y/query_progress

Watch one running query's progress

Reports how far a submitted query has got — rows scanned, bytes read, elapsed — and HOLDS the connection until the next update rather than answering immediately.

The long poll is the whole point, and the reason this cannot be a typed operation: an answer that arrived only when the query finished would report progress on nothing. The websocket form of the same read is /ws/query_progress.

A validated, org-scoped principal is required; a query id belonging to another tenant is simply not found.

POST /v1/o11y/query_range/format

Parses a builder query and echoes it back normalized to the v3 shape — the endpoint the UI uses to canonicalize a query without running it.

Callers need the viewer role; the runtime's own gate enforces it.

Request body — application/json (required)

POST /v1/o11y/query_range/preview

Validates a composite query and renders the Datastore statements it would run WITHOUT executing it — a dry run for agentic and tooling use. verbose=false trades the rendered SQL and EXPLAIN for a lightweight per-query verdict.

Callers need the viewer role; the runtime's own gate enforces it.

Request body — application/json (required)

GET /v1/o11y/query_range

Runs a Prometheus-style range query over metrics — the legacy read that predates the v5 querier — and returns the matrix, vector or scalar the query resolved to.

Callers need the viewer role; the runtime's own gate enforces it.

POST /v1/o11y/query_range

Executes a composite query over a time range: builder queries over traces, logs and metrics, formulas, trace operators, PromQL and Datastore SQL, answering time series, scalars or raw records as the request type asks.

Callers need the viewer role; the runtime's own gate enforces it.

Request body — application/json (required)

GET /v1/o11y/query

Evaluates one instant PromQL query against the org's metrics and returns the result at a single point in time.

The result is polymorphic by PromQL's own contract — a matrix, vector, scalar or string, discriminated by resultType — so it is carried verbatim rather than forced into one of its shapes.

Callers need the viewer role; the runtime's own gate enforces it.

GET /v1/o11y/readyz

Readiness of the observability runtime to serve

Reports whether the runtime's registered services are healthy enough to take traffic, and answers 503 when they are not — which is what takes a booting or degraded replica out of the load balancer instead of letting it serve errors.

UNAUTHENTICATED by design, like the other two probes. It reads the same service registry the health probe reads, so the two agree by construction; readiness is the question a router asks and health is the question an operator asks.

POST /v1/o11y/register

Creates the FIRST organization and its admin user. It is open by design — there is nobody to be signed in as yet — and refuses once setup has completed, after which new users arrive by invitation only.

Open by design; the runtime's own gate is OpenAccess.

Request body — application/json (required)

POST /v1/o11y/reset_password_tokens/verify

Checks that a reset-password token exists and has not expired, without consuming it. Unauthenticated: the token is the proof.

Request body — application/json (required)

POST /v1/o11y/resetPassword

Sets a new password for whoever the reset token was minted for, consuming the token. Unauthenticated: the token is the proof.

Request body — application/json (required)

PATCH /v1/o11y/reviews/{id}/items/{itemId}

Moves one queue item between PENDING and COMPLETED and sets its assignee. Completing an item stamps its completedAt. An item that exists under a different queue answers the same 404 an unknown item does, and so does a queue belonging to another org.

Request body — application/json (required)

GET /v1/o11y/reviews/{id}/items

Returns a page of one review queue's items, newest first, optionally filtered to PENDING or COMPLETED. A queue id belonging to another org is a 404, never a cross-tenant list.

POST /v1/o11y/reviews/{id}/items

Enqueues traces, observations or sessions on a review queue. Each item names exactly one object, either by traceId / observationId / sessionId or by objectType plus objectId; every item enters PENDING. A queue id belonging to another org is a 404.

Request body — application/json (required)

GET /v1/o11y/reviews/{id}

Returns one review queue with its pending and completed counts and its first page of items. A queue id belonging to another org is a 404, never a cross-tenant read.

PATCH /v1/o11y/reviews/{id}

Changes a review queue's name, description or score-config set. A field the request omits is left alone. A name another queue in the same project already uses is a 409; a queue id belonging to another org is a 404.

Request body — application/json (required)

DELETE /v1/o11y/reviews/{id}

Removes one review queue and every item in it. A queue id belonging to another org answers the same 404 an unknown id does, so a probe learns nothing about what exists.

GET /v1/o11y/reviews

Returns a page of the caller org's human-review queues, newest first, narrowed to the caller's project. Another org's queues are never visible.

POST /v1/o11y/reviews

Creates a human-review queue in the caller's org and project. A name already used by another queue in the same project is a 409.

Request body — application/json (required)

GET /v1/o11y/roles/{id}/users

Returns every org member holding a role, by role id. Admin gate.

GET /v1/o11y/roles/{id}

Returns one role with the transaction groups it grants.

PUT /v1/o11y/roles/{id}

Replaces a custom role's description and transaction groups. Both fields are mandatory — send an empty string or an empty array to clear one — and managed roles cannot be edited.

Request body — application/json (required)

DELETE /v1/o11y/roles/{id}

Deletes a custom role. A role that still has user or service-account assignees, or an auth-domain mapping, is refused; managed roles cannot be deleted.

GET /v1/o11y/roles

Lists every role in the caller's org — the managed ones the platform seeds and the custom ones its admins created.

POST /v1/o11y/roles

Creates a custom role in the caller's org from a name, an optional description and the transaction groups it grants, answering the new role's id.

Names are lowercase letters and hyphens only, and may not start with the reserved managed-role prefix; the runtime refuses anything else.

Request body — application/json (required)

GET /v1/o11y/route_policies/{id}

Returns one route policy, by id. Viewer gate.

PUT /v1/o11y/route_policies/{id}

Replaces a route policy, by id, answering with the stored policy. Admin gate.

Request body — application/json (required)

DELETE /v1/o11y/route_policies/{id}

Removes a route policy, by id. Admin gate.

GET /v1/o11y/route_policies

Lists the org's route policies. Viewer gate.

POST /v1/o11y/route_policies

Creates a route policy, answering with the stored policy. Admin gate.

Request body — application/json (required)

GET /v1/o11y/rules/{id}/history/filter_keys

Returns the distinct label keys present in a rule's history entries over the selected range, for building history filters. Viewer gate.

GET /v1/o11y/rules/{id}/history/filter_values

Returns the distinct values a given label key has taken across a rule's history entries. Viewer gate.

GET /v1/o11y/rules/{id}/history/overall_status

Returns the overall firing/inactive intervals for a rule over the selected range. Viewer gate.

POST /v1/o11y/rules/{id}/history/overall_status

Returns the overall firing/inactive windows for a rule, for the posted query range. Viewer gate.

Request body — application/json (required)

GET /v1/o11y/rules/{id}/history/stats

Returns trigger and resolution statistics for a rule over the selected time range, current window against the prior one. Viewer gate.

POST /v1/o11y/rules/{id}/history/stats

Returns trigger and resolution statistics for a rule, current window against the prior one, for the posted query range. Viewer gate.

Request body — application/json (required)

GET /v1/o11y/rules/{id}/history/timeline

Returns paginated timeline entries for a rule's state transitions, filterable by state and a label expression, cursor-paginated. Viewer gate.

POST /v1/o11y/rules/{id}/history/timeline

Returns a rule's state-transition timeline for the posted query range, each entry carrying its related-logs or related-traces link. Viewer gate.

Request body — application/json (required)

GET /v1/o11y/rules/{id}/history/top_contributors

Returns the label combinations that contributed most to a rule firing over the selected range. Viewer gate.

POST /v1/o11y/rules/{id}/history/top_contributors

Returns the label combinations that contributed most to a rule firing, for the posted query range. Viewer gate.

Request body — application/json (required)

GET /v1/o11y/rules/{id}

Returns one alert rule with its evaluation state, by id. Viewer gate.

PUT /v1/o11y/rules/{id}

Replaces an alert rule's definition, by id. Editor gate.

PATCH /v1/o11y/rules/{id}

Applies a partial update to an alert rule, by id, answering with the stored rule — the common toggle for enabling or muting a rule. Editor gate.

DELETE /v1/o11y/rules/{id}

Removes an alert rule, by id. Editor gate.

POST /v1/o11y/rules/test

Fires a test notification for a rule definition without saving it, answering with how many series would alert. Editor gate.

GET /v1/o11y/rules

Lists all alert rules with their current evaluation state. Viewer gate.

POST /v1/o11y/rules

Creates a new alert rule and answers with the stored rule. Editor gate.

PUT /v1/o11y/service_accounts/{id}/keys/{fid}

Renames an API key or moves its expiry.

Request body — application/json (required)

DELETE /v1/o11y/service_accounts/{id}/keys/{fid}

Revokes an API key. Revocation is immediate and permanent.

GET /v1/o11y/service_accounts/{id}/keys

Lists a service account's API keys — metadata only, never the secrets.

POST /v1/o11y/service_accounts/{id}/keys

Mints an API key for a service account and answers the key's id and its secret — the one time the secret is ever shown.

ExpiresAt is a unix timestamp in seconds; zero means the key never expires, and a timestamp in the past is refused.

Request body — application/json (required)

DELETE /v1/o11y/service_accounts/{id}/roles/{rid}

Removes a role from a service account.

GET /v1/o11y/service_accounts/{id}/roles

Lists the roles a service account holds.

POST /v1/o11y/service_accounts/{id}/roles

Assigns a role, named by its id, to a service account.

Request body — application/json (required)

GET /v1/o11y/service_accounts/{id}

Returns one service account with the roles it holds.

PUT /v1/o11y/service_accounts/{id}

Renames a service account.

Request body — application/json (required)

DELETE /v1/o11y/service_accounts/{id}

Deletes a service account and revokes every key it holds.

GET /v1/o11y/service_accounts/me

Returns the calling service account itself, with the roles it holds — the self-inspection read for a key-authenticated caller.

PUT /v1/o11y/service_accounts/me

Renames the calling service account.

Request body — application/json (required)

GET /v1/o11y/service_accounts

Lists the caller's org's service accounts.

POST /v1/o11y/service_accounts

Creates a service account in the caller's org, answering its id. The name — a lowercase letter followed by lowercase letters, digits or hyphens — becomes the account's email local part.

Request body — application/json (required)

POST /v1/o11y/service/entry_point_operations

Returns one service's entry-point operations with the same latency and error profile topOperations reports.

Request body — application/json (required)

POST /v1/o11y/service/top_level_operations

Maps each service to its entry-point span names — for the one service named in the request, or for every service when none is.

Request body — application/json (required)

POST /v1/o11y/service/top_operations

Returns one service's heaviest operations in the window, each with p50/p95/p99 latency, how often it ran and how often it errored.

Request body — application/json (required)

GET /v1/o11y/services/list

Lists the name of every service the trace store holds, with no window applied — the complete catalog, for pickers and autocomplete.

POST /v1/o11y/services

Lists the instrumented services seen in the window, each with the request profile of its entry-point spans: p99 and average latency, call and error rates, and the entry-point operations the numbers were computed over.

Request body — application/json (required)

GET /v1/o11y/sessions/context

Tells a sign-in page what an email address can do: which orgs the address belongs to and, per org, which password and SSO routes are open to it. Unauthenticated: it runs before any session exists.

POST /v1/o11y/sessions/email_password

Signs a user in with email and password and answers with the session's token pair. Unauthenticated: this call is how authentication begins.

Request body — application/json (required)

POST /v1/o11y/sessions/rotate

Exchanges a refresh token for a fresh token pair, retiring the old pair. The access token being rotated identifies the session.

Request body — application/json (required)

GET /v1/o11y/sessions

List the caller org's LLM sessions

Answers the caller org's LLM-observability sessions — traces grouped by session id on the gen_ai span plane — paged by limit and offset, in the runtime's own envelope, passed through unchanged.

An org-less caller is refused HERE, at the cloud boundary, before the request reaches the runtime, and the org the runtime then scopes on is that SAME validated tenant. The two cannot disagree: the tenant is minted from the principal's own claim at ingress and a client copy never survives it.

There is deliberately no session-detail route to pair with this. The runtime serves the list only; detail is composed client-side from this list plus the traces filtered by session, so a caller looking for one is looking for something that was never served rather than something that broke.

DELETE /v1/o11y/sessions

Signs the calling session out, invalidating its tokens. The access token on the call names the session to end.

GET /v1/o11y/settings/apdex

Returns apdex settings for the named services.

Callers need the viewer role; the runtime's own gate enforces it.

POST /v1/o11y/settings/apdex

Sets one service's apdex threshold and the status codes excluded from its score.

Admin only, as the mux tree has always gated it (AdminAccess); the runtime's own gate enforces it.

Request body — application/json (required)

GET /v1/o11y/settings/ttl

Returns the org's current retention policy: default TTL, custom per-label rules, and cold-storage settings where configured.

Callers need the viewer role; the runtime's own gate enforces it.

POST /v1/o11y/settings/ttl

Sets the org's retention policy for one signal: the default TTL in days, ordered per-label retention rules, and optional cold-storage settings.

Admin only, as the mux tree has always gated it (AdminAccess); the runtime's own gate enforces it.

Request body — application/json (required)

PATCH /v1/o11y/span_mapper_groups/{groupId}/span_mappers/{mapperId}

Changes a mapper's field context, config or enabled state. Every field is optional and only the ones sent are applied.

Callers need the admin role; the runtime's own gate enforces it.

Request body — application/json (required)

DELETE /v1/o11y/span_mapper_groups/{groupId}/span_mappers/{mapperId}

Deletes one mapper from a group.

Callers need the admin role; the runtime's own gate enforces it.

GET /v1/o11y/span_mapper_groups/{groupId}/span_mappers

Lists the mappers belonging to one group, in the order they are applied.

Callers need the viewer role; the runtime's own gate enforces it.

POST /v1/o11y/span_mapper_groups/{groupId}/span_mappers

Adds a mapper to a group: which field context it reads, the move or copy it performs, and whether it is on.

Callers need the admin role; the runtime's own gate enforces it.

Request body — application/json (required)

PATCH /v1/o11y/span_mapper_groups/{groupId}

Changes a group's name, condition or enabled state. Every field is optional and only the ones sent are applied.

Callers need the admin role; the runtime's own gate enforces it.

Request body — application/json (required)

DELETE /v1/o11y/span_mapper_groups/{groupId}

Deletes a mapping group and every mapper under it.

Callers need the admin role; the runtime's own gate enforces it.

GET /v1/o11y/span_mapper_groups

Lists the caller's org's mapping groups, optionally only the enabled ones.

Callers need the viewer role; the runtime's own gate enforces it.

POST /v1/o11y/span_mapper_groups

Creates a mapping group: the name it is known by, the span and resource attributes whose presence selects a span into it, and whether it is on.

Callers need the admin role; the runtime's own gate enforces it.

Request body — application/json (required)

POST /v1/o11y/span_percentile

Places one span's duration among its peers: the p50/p90/p99 durations of like spans, and the percentile the given duration lands at.

Callers need the viewer role; the runtime's own gate enforces it.

Request body — application/json (required)

GET /v1/o11y/statefulsets/attribute_keys

Lists the metric attribute keys Kubernetes statefulsets report, for building statefulset filters.

GET /v1/o11y/statefulsets/attribute_values

Lists the values one statefulset attribute key has taken, for building statefulset filters.

POST /v1/o11y/statefulsets/list

Lists Kubernetes statefulsets over a time range, each with the CPU and memory its pods used against request and limit, desired and available replica counts, restarts and attributes; filterable, groupable and paginated.

Request body — application/json (required)

GET /v1/o11y/stats

Returns the collected usage statistics for the caller's org, as the stats reporter aggregates them — a map whose keys are the reporter's own counter names.

Callers need the viewer role; the runtime's own gate enforces it.

GET /v1/o11y/status

Reports whether a product's service is live: an in-cluster health probe with its measured latency, fused with the per-replica up inventory. Infra health is not tenant-partitioned — a service is up or down for everyone — so any validated caller is served, but an unvalidated one is refused. A product with no backing workload answers down/unknown-service without probing anything; a malformed slug is a 400.

POST /v1/o11y/substitute_vars

Substitutes a query's variables and returns the resolved request, without running it — what a dashboard does before it queries.

Callers need the viewer role; the runtime's own gate enforces it.

Request body — application/json (required)

POST /v1/o11y/testChannel

Sends a test notification to the posted receiver. The legacy path; prefer /channels/test. Editor gate.

Request body — application/json (required)

POST /v1/o11y/testRule

Fires a test notification for the posted rule definition and answers with how many series alerted and a status message. The legacy path; prefer /rules/test. Editor gate.

POST /v1/o11y/third-party-apis/overview/domain

Returns one external domain's endpoint-level breakdown — each endpoint with its rate, error and latency columns over the window.

Request body — application/json (required)

POST /v1/o11y/third-party-apis/overview/list

Lists the external domains the instrumented services call, with request rate, error percentage and latency per domain. Rows whose domain is a bare IP address are dropped unless show_ip asks for them.

Request body — application/json (required)

POST /v1/o11y/trace-funnels/{funnel_id}/analytics/error-traces

Returns the errored traces through a step transition of a saved funnel — the entry point for "why is this step failing".

Request body — application/json (required)

POST /v1/o11y/trace-funnels/{funnel_id}/analytics/overview

Returns a saved funnel's conversion overview over a window: how many entered, how many converted, the rate and the latency.

Request body — application/json (required)

POST /v1/o11y/trace-funnels/{funnel_id}/analytics/slow-traces

Returns the slowest traces through a step transition of a saved funnel — the entry point for "why is this step slow".

Request body — application/json (required)

POST /v1/o11y/trace-funnels/{funnel_id}/analytics/steps/overview

Returns the conversion between two named steps of a saved funnel — the step-to-step drill-down behind the overview.

Request body — application/json (required)

POST /v1/o11y/trace-funnels/{funnel_id}/analytics/steps

Returns a saved funnel's per-step metrics over a window — the counts and latencies at each step, in step order.

Request body — application/json (required)

POST /v1/o11y/trace-funnels/{funnel_id}/analytics/validate

Lists the traces that match a saved funnel over a window — the read that answers "is this funnel finding anything at all".

Request body — application/json (required)

GET /v1/o11y/trace-funnels/{funnel_id}

Returns one funnel with its steps.

Callers need the viewer role; the runtime's own gate enforces it.

PUT /v1/o11y/trace-funnels/{funnel_id}

Renames a funnel or rewrites its description, answering the funnel as it now stands.

Callers need the editor role; the runtime's own gate enforces it.

Request body — application/json (required)

DELETE /v1/o11y/trace-funnels/{funnel_id}

Deletes a funnel. The answer carries no data — the runtime acknowledges with the success envelope alone, which is what this Out says.

Callers need the editor role; the runtime's own gate enforces it.

POST /v1/o11y/trace-funnels/analytics/error-traces

Returns the errored traces through a step transition of a funnel described inline.

Request body — application/json (required)

POST /v1/o11y/trace-funnels/analytics/overview

Returns the conversion overview of a funnel described inline.

Request body — application/json (required)

POST /v1/o11y/trace-funnels/analytics/slow-traces

Returns the slowest traces through a step transition of a funnel described inline.

Request body — application/json (required)

POST /v1/o11y/trace-funnels/analytics/steps/overview

Returns the conversion between two steps of a funnel described inline.

Request body — application/json (required)

POST /v1/o11y/trace-funnels/analytics/steps

Returns the per-step metrics of a funnel described inline.

Request body — application/json (required)

POST /v1/o11y/trace-funnels/analytics/validate

Lists the traces that match a funnel described inline — the builder's "try this" before anything is saved.

Request body — application/json (required)

GET /v1/o11y/trace-funnels/list

Lists the caller's org's funnels, each with its steps and who last touched it.

Callers need the viewer role; the runtime's own gate enforces it.

POST /v1/o11y/trace-funnels/new

Creates an empty funnel with a name, answering the funnel it created. Steps are added afterwards with the steps update.

Callers need the editor role; the runtime's own gate enforces it.

Request body — application/json (required)

PUT /v1/o11y/trace-funnels/steps/update

Replaces a funnel's steps — the funnel is named in the body rather than the path — and answers the funnel as it now stands. A name or description sent alongside is applied too; an empty one leaves it as it was.

Callers need the editor role; the runtime's own gate enforces it.

Request body — application/json (required)

POST /v1/o11y/traces/{traceId}/aggregations

Computes span aggregations over one trace — span count, duration or share of execution time — grouped by the resource field each aggregation names.

Callers need the viewer role; the runtime's own gate enforces it.

Request body — application/json (required)

POST /v1/o11y/traces/{traceId}/flamegraph

Returns a trace's flamegraph: spans bucketed by depth level, each level ordered as it is drawn, around the selected span.

Callers need the viewer role; the runtime's own gate enforces it.

Request body — application/json (required)

POST /v1/o11y/traces/{traceId}/waterfall

Returns a trace's waterfall: every span when the trace is small enough, a capped window around the selected span when it is not, with the uncollapsed subtrees the caller asked to keep open.

Callers need the viewer role; the runtime's own gate enforces it.

Request body — application/json (required)

GET /v1/o11y/traces/{traceId}

Returns one trace's spans as a column/row table, optionally centred on a span and walked a fixed number of levels up and down from it — the read the trace explorer opens a trace with.

Callers need the viewer role; the runtime's own gate enforces it.

GET /v1/o11y/traces/fields

Returns the trace field catalog: the span fields already selected as indexed columns, and the interesting ones seen in the data that could be.

Callers need the viewer role; the runtime's own gate enforces it.

POST /v1/o11y/traces/fields

Changes how one span field is stored — selects or deselects it as a materialized column and tunes its index — and echoes the setting back.

Callers need the editor role; the runtime's own gate enforces it.

Request body — application/json (required)

GET /v1/o11y/traces

Lists the caller org's recent traces — one row per trace with its span count and wall-clock duration, most recently active first. This is the trace SEARCH: it is where a trace id comes from, and the spans behind any row are then read from GET /v1/o11y/traces/{traceId}. Every row belongs to the caller's own org — the tenant is the validated principal, never an input, and there is no administrator widening, because a trace list is a tenant's records rather than a rollup over them. An unreachable telemetry store answers 503 rather than an empty page, because "no traces" and "cannot see the traces" are different facts and only one of them is about the caller's system.

GET /v1/o11y/usage

Returns ingestion usage counts bucketed over the requested window, optionally narrowed to one service.

Callers need the viewer role; the runtime's own gate enforces it.

GET /v1/o11y/user/{id}

Returns one org member with their single legacy role, by user id. Admins may read anyone; a non-admin only themselves (the runtime's self-access gate).

PUT /v1/o11y/user/{id}

Renames one org member and may move their legacy role, answering with the updated record. Admins may update anyone; a non-admin only themselves (the runtime's self-access gate).

Request body — application/json (required)

DELETE /v1/o11y/user/{id}

Removes one org member, by user id. The same operation as deleteUser on the legacy singular path. Admin gate.

GET /v1/o11y/user/me

Returns the calling user with their single legacy role. Deprecated in favor of getMyUser. Open to any authenticated caller.

GET /v1/o11y/user/preferences/{name}

Returns one preference of the calling user, by name. Viewer gate.

PUT /v1/o11y/user/preferences/{name}

Sets one preference of the calling user, by name. Viewer gate.

Request body — application/json (required)

GET /v1/o11y/user/preferences

Lists every preference of the calling user, each with its current and default value. Viewer gate.

GET /v1/o11y/user

Lists the org's members with their single legacy role. Deprecated in favor of listUsers, which answers without the role. Admin gate.

GET /v1/o11y/users/{id}/reset_password_tokens

Returns the reset-password token a user already has; absent one, the answer is a not-found rather than a fresh token. Admin gate.

PUT /v1/o11y/users/{id}/reset_password_tokens

Creates or regenerates a user's reset-password token: a live token is returned as it is, an expired one is replaced. Admin gate.

DELETE /v1/o11y/users/{id}/roles/{roleId}

Takes a role away from one org member, by user id and role id — someone else, never the caller. Admin gate.

GET /v1/o11y/users/{id}/roles

Returns every role one org member holds, by user id. Admin gate.

POST /v1/o11y/users/{id}/roles

Assigns a role, by role name, to one org member — someone else, never the caller. Admin gate.

Request body — application/json (required)

GET /v1/o11y/users/{id}

Returns one org member together with every role they hold, by user id. Admin gate.

PUT /v1/o11y/users/{id}

Renames one org member, by user id — someone else, never the caller, who renames themselves through updateMyUser. Admin gate.

Request body — application/json (required)

DELETE /v1/o11y/users/{id}

Removes one org member, by user id. Admin gate.

PUT /v1/o11y/users/me/dashboards/{id}/pins

Pins a dashboard for the calling user. A user can pin at most ten dashboards; pinning at the limit refuses with the runtime's conflict. Re-pinning an already-pinned dashboard is a no-op success. Pinning mutates only the caller's pin list, not the dashboard, so a viewer may pin what a viewer may read.

Callers need the viewer role; the runtime's own gate enforces it.

DELETE /v1/o11y/users/me/dashboards/{id}/pins

Removes the caller's pin for a dashboard. Idempotent — unpinning a dashboard that was not pinned still succeeds.

Callers need the viewer role; the runtime's own gate enforces it.

GET /v1/o11y/users/me/dashboards

Is dashboardListV2 personalized for the calling user: each dashboard carries the caller's pinned state, and pinned dashboards float to the top of the requested ordering. Supports the same filter DSL, sort, order and pagination.

Callers need the viewer role; the runtime's own gate enforces it.

PUT /v1/o11y/users/me/factor_password

Replaces the calling user's password, refusing when the old one does not match. Open to any authenticated caller.

Request body — application/json (required)

GET /v1/o11y/users/me

Returns the calling user together with every role they hold. Open to any authenticated caller.

PUT /v1/o11y/users/me

Renames the calling user. Open to any authenticated caller.

Request body — application/json (required)

GET /v1/o11y/users

Lists the caller's org members. Admin gate.

POST /v1/o11y/users

Creates a member of the caller's org in the pending-invite state and mails them their invitation; the answer is the new user's id. Admin gate.

Request body — application/json (required)

POST /v1/o11y/variables/query

Evaluates a dashboard variable query and returns the values the variable may take.

Callers need the viewer role; the runtime's own gate enforces it.

Request body — application/json (required)

GET /v1/o11y/version

Reports the running build: its version, whether an enterprise edition is present ("N" in this build), and whether first-user setup has completed.

Open by design; the runtime's own gate is OpenAccess.

GET /v1/o11y/zeus/hosts

Returns the deployment's host info from Zeus. Viewer gate.

PUT /v1/o11y/zeus/hosts

Records the deployment's host in Zeus, overwriting any prior one. Admin gate.

Request body — application/json (required)

PUT /v1/o11y/zeus/profiles

Records the deployment's profile in Zeus — how the team uses observability today and what they plan — overwriting any prior one. Admin gate.

Request body — application/json (required)

---

O11y guide · All Hanzo APIs · Interactive reference