Pick the model your project
should actually ship.
Public benchmarks rank models on someone else's task. Agent Arena ranks them on yours — your test cases, your accuracy floor, your budget, your latency ceiling — and refuses to recommend anything you could not ship.
pip install agent-arena && arena ui
| # | model | composite | accuracy | cost | latency | status |
| 1 | sim_small | 0.853 | 83.3% | $0.06 | 190ms | ranked |
| 2 | sim_frontier | 0.750 | 97.6% | $0.30 | 1,500ms | ranked |
| 3 | sim_balanced | 0.744 | 85.7% | $0.18 | 620ms | ranked |
| — | sim_tiny | — | 50.0% | $0.02 | 90ms | DISQUALIFIED |
What this does that a benchmark leaderboard does not
MMLU and public arenas tell you how a model does on tasks you did not choose, averaged over criteria you did not set. That is close to useless for a shipping decision, because your accuracy floor, latency budget and price ceiling are yours.
You define what “best” means
Weight accuracy, cost and latency however your product actually works. A chatbot people wait on and an overnight batch job are different questions, and they get different answers from the same data.
Unusable models are disqualified, not ranked
Set a hard floor and a model that misses it comes back DISQUALIFIED with the reason printed. A leaderboard that quietly ranks an unshippable model fourth is lying to you.
It admits when it cannot tell
When the top two are within 0.02 the report says the sweep cannot separate them and asks for more cases — instead of implying a twelve-case run settled it.
Every claim about an LLM system is about one of two variables
When a multi-agent pipeline fails, was it the model being dumb or the architecture dropping data on the floor? You cannot answer that by varying both at once. So each half pins one and moves the other.
Universal Arena varies the model
A project is a folder: config.yaml + tests.yaml. Ten built-in
scorers, hard constraints, a weighted composite, and every run stored in SQLite so you
can track regressions across releases. Runs offline against simulated models before you
spend anything.
Handoff study varies the architecture
A frozen 28-run experiment holding the model fixed and swapping the coordination pattern. It emits a replayable trace and grades every run into a failure category, so a coordination failure can be told apart from a capability failure.
Read the study →Splitting a task across agents loses information at the handoff
Same model throughout. The only variable was the wiring. One fact — is this account on credit hold? — is seen by the first stage and needed by the last.
| architecture | accuracy | status |
| single_agent | 100.0% | ranked |
| peer_to_peer | 27.3% | DISQUALIFIED |
| supervisor_worker | 63.6% | DISQUALIFIED |
From projects/pipeline_demo — three architectures, one task, fully offline.
- ▸ A rigid handoff loses it every time. The fixed-format string has no field for the flag, so the deciding stage never sees it. No better model fixes that — the data is gone before the model is asked anything.
- ▸ A free-text summary loses it sometimes. The fact survives when it is prominent and vanishes when it is buried late in a long input. Intermittent is the worse bug: it reads like a prompt problem for weeks.
- ▸ More agents cost more. Each extra hop is real money and real latency, and the leaderboard prices it — so “add a critic agent” becomes a decision with a number attached.
A real leaderboard in about a minute, with no API key
Both example projects run offline against deterministic simulated models.
pip install -e .
# see a real leaderboard immediately — no key needed
arena evaluate --project projects/support_triage
# compare multi-agent architectures instead of models
arena evaluate --project projects/pipeline_demo
# or drive the whole thing from a browser
arena uiThen your own project
arena init projects/my_project
arena validate --project projects/my_project
arena evaluate --project projects/my_project --dry-run
arena evaluate --project projects/my_project
Two files decide everything: config.yaml holds the models, weights and hard
constraints; tests.yaml holds your cases. There is no second code path and
no plugin to write.
Read the details
Universal Arena guide
Projects, scorers, metrics, hard constraints, pipeline targets and the browser UI.
Open →Multi-agent handoff study
The experiment, the four architectures and the committed 28-run sweep.
Open →Demo walkthrough
End-to-end against models running on your own machine, with real output.
Open →Sample report
Exactly what a finished run writes to disk.
Open →Roadmap
What ships next — and what this deliberately will not become.
Open →Decisions (ADRs)
Why the trace format, retry strategy and config-driven design are what they are.
Open →