Observability¶

vornik is built to run unattended, so it gives you several ways to see what it's doing: a set of operator dashboards in the Web UI, and a Prometheus metrics endpoint for your own monitoring stack.

The dashboards¶

Home¶

The Web UI landing page (/ui/) is an operator front page of live tiles: active tasks (queued, leased, running), active chats, a countdown to the next autonomy evaluation per project, the response cache's savings (calls Vornik answered itself, across all cached entries), a memory pipeline snapshot, and an Inbox tile for anything that needs you.

Insight¶

The Insight area collects the analytical views:

Spend (/ui/spend) — the cost deep-dive (below).
Trends (/ui/insights/trends) — daily throughput and success-rate over a trailing window, plus a recovery-events series when recovery tracking is wired.
Tool budget (/ui/insights/tool-budget) — a histogram of how many tool calls executions actually use, plotted against the complexity-tier budget, with mean / P50 / P95. Use it to tune budgets to reality.

Health¶

Admin-gated health tabs under /ui/admin/health/ cover the daemon's operational state: leases (lease status and recent lease transitions — the surface for diagnosing stuck or lost work), the stuck-execution watchdog, MCP server reachability, runtime probes, and cluster leader/heartbeat status.

Spend and cost-efficiency¶

The spend dashboard (/ui/spend) slices cost by window (24h / 7d / 30d), project, source, task, role+model, and API key. Alongside total cost and token counts it surfaces a few efficiency signals.

Embedding spend appears under the role Memory · Embedder and the source Memory embedding. It is included in every rollup on the page — a deployment using a hosted embedder can see what ingest and retrieval cost, not only what chat cost. The LLM reranker that reorders recall candidates appears as Memory · Reranker (source Memory reranking).

Input ratio — prompt tokens as a share of total; a persistently high ratio flags context bloat.
Cache hit ratio and dollars saved — how much prompt/embedding caching is actually saving.
Estimated token counts — a ~ before a token figure means at least one call behind that line reported no token count, so the total is partly derived from text length rather than measured by the provider. Embedding backends differ here: an OpenAI-compatible endpoint reports usage and Bedrock Titan reports an input-token count, while Bedrock Cohere reports neither. The marker exists so an inferred number is never mistaken for a measured one when reconciling a provider bill.
Effective cost per success and its drift ratio — cost per useful result (spend divided by successful outcomes), and how the recent window compares to a baseline. A cheap model that fails often can cost more per success than an expensive one that doesn't; the drift badge (green/amber/red) flags when that's getting worse. See Cost and caching.

Spend per API key¶

The Spend per API key table attributes LLM cost to the key that authenticated the call, so you can see which integration is spending — a companion session, a CI caller, a script hitting POST /tasks.

Read it as per key, not per person. A key is minted per (project, client, session), so one teammate routinely holds several and each shows as its own row; plenty of keys have no person behind them at all. Vornik's user accounts (admin UI logins, sessions, approvals) are a separate concept and are not what this table groups by.

Rows land in Unattributed when the spend has no reliable key behind it: dispatcher chat, memory background workers, the project wizard, UI authoring assist, and the fix-it doctor. Those are shown as one bucket rather than guessed at. Attribution starts at the migration that added it — spend recorded before then stays unattributed, with no backfill.

A key's own spend cap (vornikctl companion grant --budget-usd) is a narrower number than its row here: the cap counts spend on tasks the key created, while the table also includes task-less calls the OpenAI-compatible chat proxy attributes to it. Both resolve the key the same way; they differ only in which rows they count.

Upgrade note: caps may start binding where they didn't

Before per-key attribution existed, the cap could only recognise a key through the marker left by companion delegate, so tasks a key created through the REST API (POST /tasks) were invisible to it — that spend was effectively uncapped. After upgrading, both paths count. A key whose cap looked like it had headroom may be at or over it, and its next delegate will be refused with BUDGET_EXCEEDED. Nothing is retroactively billed; only the gate's view widened. If you rely on caps, check vornikctl companion list against the per-key rows on /ui/spend before upgrading and raise any cap that was sized against companion traffic alone.

From the CLI, vornikctl cache-stats reports cache effectiveness. (Note: vornikctl cpc is the cross-project-call ledger, not a cost command — there is no vornikctl spend.)

Prometheus metrics¶

vornik exposes Prometheus metrics at /metrics. Enable the dedicated metrics server and point your scraper at it:

metrics:
  enabled: true
  addr: ":9090"            # dedicated metrics listener; scrape http://<host>:9090/metrics
  require_admin: false     # when true (with auth on), the main-port /metrics needs a token
  # scrape_token: "…"      # a read-only credential accepted alongside the admin key

/metrics is also mounted on the main API port. When you run with authentication on and set require_admin: true, the main-port endpoint requires the admin key or a dedicated scrape token; the separate metrics.addr listener is the intended scrape target — bind it to a trusted interface.

Every /metrics that answers without a credential (the dedicated listener always, and the main-port endpoint unless require_admin is set with auth on) serves only requests addressed to an IP address, localhost, the daemon's public base URL host, or a name listed in server.accepted_hosts. This stops a web page from reading your metrics through DNS rebinding. If your scraper reaches vornik by another name (for example a compose service name such as vornik:9090), add that name to server.accepted_hosts; otherwise the scrape gets 403.

The exported metrics cover, in plain terms:

Task and execution lifecycle — active, started, completed, failed, cancelled, retried counts, and durations.
Queue, scheduling, and leases — queue depth by status, how long tasks wait before running, leases acquired and expired, and recoveries of expired leases.
LLM cost and tokens — dollar spend per project and per model, prompt and completion tokens, tool-calling iterations, and cache savings.
Model quality — per role and model: success rate, effective cost per success, and rates of parse failures, schema violations, refusals, and degenerate loops.
Tool calls, autonomy, and cross-project orchestration — tool invocation counts, autonomy approval resolutions and configured caps, cross-project call and spawn counts, and outcomes of agent-recorded backlog deposits (accepted, or rejected as a duplicate, a secret hit, over the per-task cap, or in cooldown).
Scheduled updates (task-kind reminders) — task-kind fires per project, outcome deliveries by success/failure, fires skipped because the prior run was still going, delivery-time channel-send failures, outcomes that had nowhere to go (no channel resolver, or the channel not configured), and stuck-firing rows the crash-recovery sweep reclaims. See the vornikctl reference (reminders group) for the text vs task kinds and pause/resume.
HTTP and process — request counts, durations, in-flight requests, plus standard Go/process collectors.

OpenTelemetry tracing can be enabled alongside metrics with tracing.enabled and tracing.endpoint.

See the Configuration reference for every metrics.* key and default.