Skip to content

Run a model comparison

Compare two models on Vornik's agent benchmark: the same tasks, the same harness, two arms, and a scoreboard that refuses to compare what is not comparable. How the benchmark works and what it does not measure: How the agent benchmark works.

A full pass is long. The published agent results ran 30 tasks × 3 repeats in about 10 hours per arm on one GPU (a median of about 5 minutes per run). A release gate needs 10 repeats per task, so plan for days, not hours. Start with --smoke.

1. A bench daemon, not your production one

Work from a checkout of the repository and build the CLI from it: go build -o /tmp/vornikctl ./cmd/vornikctl (the script says so if the vornikctl it finds lacks bench agent).

The harness writes tasks, workspaces and memory. Run a second Vornik daemon for it, with its own database, listener and config tree holding only autonomy-off projects. The script's defaults expect that daemon on port 8090 with its config in ~/.config/vornik-bench/; --daemon-config, --daemon-binary and --workspace point it elsewhere.

export VORNIK_URL=http://127.0.0.1:8090        # the bench daemon
export VORNIK_API_URL="$VORNIK_URL"            # vornikctl reads this one
export VORNIK_COMPANION_TOKEN=<a key from: vornikctl companion grant -p agentbench --client claude-code ...>
export VORNIK_BENCH_DSN="host=... dbname=... user=... password=... sslmode=disable"   # the bench database

The script refuses to run against a database other than the one the bench daemon writes.

2. Prove the pipeline

scripts/agentbench-reproduce.sh --smoke

One task, one run. Fix whatever it reports before spending a full pass.

3. Write a pre-registration

Copy one from preregistrations/ in the agentbench package of the repository's internal directory. Edit it before you run anything: the two arms, the metric, the difference worth detecting (δ) and the repeats it needs. The harness records its hash in every journal, and the release gate's calibrate and noise-floor refuse a journal whose pre-registration was edited after the run.

4. Run both arms

Configure the bench daemon's agent model for the first arm, then run it; change the model, restart the bench daemon, run the second. Each run writes its journals under ./agentbench-runs (--out moves it):

scripts/agentbench-reproduce.sh --arm baseline  --preregistration my-prereg.json --repeats 10
scripts/agentbench-reproduce.sh --arm candidate --preregistration my-prereg.json --repeats 10

--repeat-batch splits a long pass into chunks that journal separately, and --budget-tokens caps what a pass may spend against a prepaid provider. The script's header lists every option.

5. Read the result

vornikctl bench agent report  <journal>               # one arm's scoreboard
vornikctl bench agent rollup  <journal>...            # cost, efficiency, accuracy, success rate
vornikctl bench agent compare <journal-a> <journal-b> # the difference, or a refusal

compare refuses two arms that differ in more than the axis you pre-registered. A refusal is a result: it says the runs measured different things.

What a comparison does not tell you

  • It measures these tasks, with this harness, on your machine.
  • A task set published before a model was trained may be in its training data; prefer the newest set for a decision.
  • Fewer than 10 repeats gives no usable σ; the gate refuses to decide on it.