Cost control

An evaluation is a spending event. Four mechanisms keep it deliberate.

1. Prove it offline first

mock: models have fixed accuracy, latency and price, and run with no network and no credential.

models:
  - key: sim_small
    model: mock:small
    card: {input_usd_per_mtok: 1, output_usd_per_mtok: 5}

arena evaluate --project projects/my_project --models sim_small

Prove your scorer grades correctly, your weights rank the way you intend, and your constraints fire — all before spending anything. A scorer bug found on the first real run has already cost you the whole sweep.

This is the single most effective cost control in the tool, and it is free.

2. Plan before you spend

arena evaluate --project projects/my_project --dry-run

Prints the matrix and an estimate; calls nothing. models × cases × trials grows multiplicatively — four models, 50 cases and 3 trials is 600 calls, which is easy to type and expensive to discover afterwards.

3. Know what a call costs

Cost comes from the price book (model_cards.json), a project pricing file, or a per-model card: override.

models:
  - key: my_model
    model: some-model
    card: {input_usd_per_mtok: 3.0, output_usd_per_mtok: 15.0}

The all-or-nothing rule

If any completed call lacks a price, the cost metric is None for the whole run. Not a partial mean — nothing.

That is deliberate. A mean over the priced subset would read as "this model costs X" when it means "the part we could price costs X", and a purchasing decision made on that number would be wrong in an invisible way.

The practical consequence: adding one unpriced model removes the cost axis for every model in the run. The leaderboard warns when this happens and names the unpriced models. Fix it with a card: override, a pricing file, or by having a run: target report its own cost_usd.

Prices move. The catalog records as_of; verify against your provider before making a decision on cost.

4. Limit the sweep

arena evaluate --project p --tags smoke --trials 1 --limit 20

Budgets

budgets:
  max_run_usd: 5.00
  max_model_usd: 2.00
  confirm_above_usd: 1.00
  on_exceed: stop

Enforced during the run. Cost is accumulated as each call completes, and when a cap is crossed the sweep stops. The results already collected are kept and the leaderboard is labelled partial:

Run stopped early (budget: spent $0.0240, over the $0.02 limit for this run).
These results cover 4 of 60 planned calls and are partial.

Keeping the partial results is deliberate: the calls were already paid for, so discarding their answers wastes the spend without saving any.

on_exceed: warn records the overrun without stopping. confirm_above_usd is parsed but not yet used to prompt.

Constraints are not a cost control, but they help

constraints:
  max_cost_per_1k_calls_usd: 2.0

This disqualifies a model that is too expensive after the run — it does not prevent the spend. Its value is stopping an expensive model from winning, which is a different and also useful thing.

Watching a run

The progress line shows completed versus planned, and the summary prints total spend:

  4 models × 12 tests × 3 trial(s) = 144 calls in 0.0s
  Spend: $0.0199

A run in progress can be cancelled: POST /api/jobs/{job_id}/cancel. The runner checks between calls — not inside one, because a request already sent has already been paid for — so the stop takes effect within one call per worker. The partial results are kept and labelled, exactly as with a budget stop.

Ctrl-C also works from the terminal and closes the run out as aborted with its partial results attached.

Reviewing spend afterwards

sqlite3 projects/my_project/results/arena.sqlite \
  "SELECT run_id, started_at, total_cost_usd FROM runs ORDER BY started_at DESC LIMIT 10;"

sqlite3 projects/my_project/results/arena.sqlite \
  "SELECT model_key, COUNT(*), ROUND(SUM(cost_usd), 4)
   FROM results WHERE run_id = '<id>' GROUP BY model_key;"

There is no cross-project spend view yet.

A safe first paid run

arena evaluate --project p --models sim_small          # 1. offline, free
arena validate --project p                             # 2. credentials present
arena evaluate --project p --dry-run                   # 3. see the plan
arena evaluate --project p --limit 5 --trials 1        # 4. tiny real run
arena evaluate --project p                             # 5. the real thing

Step 4 is the one people skip. Five real calls will surface a malformed request, a wrong model id or a broken scorer for a few cents.