support_triage — model evaluation
What a finished run actually produces.
Generated by arena evaluate --project projects/support_triage and committed as a sample of the output. The models are the project's simulated ones, so this runs offline and reproduces exactly.
Route inbound support tickets to one of six queues. Volume is high and the answer is one word, so cost and latency matter nearly as much as accuracy — but a misroute is expensive, so anything under 70% accuracy is unusable.
Recommendation
sim_small — Small/fast (simulated) (mock:small) — composite 0.853. accuracy 83.3%, cost $0.06, latency 190ms
It beats sim_frontier by 0.103 on the composite, while losing on accuracy (83.3% vs 97.6%).
Run
| field | value |
|---|---|
| run id | run_20260806_174425_c8a317 |
| generated | 2026-08-06 17:44 UTC |
| models | 4 |
| test cases | 12 |
| trials per case | 3 |
| calls | 144 |
| failed calls | 0 |
| wall clock | 0.1s |
| total spend | $0.0199 |
| weights | accuracy 55%, cost 25%, latency 20% |
Leaderboard
| # | model | id | composite | accuracy | cost | latency | status |
|---|---|---|---|---|---|---|---|
| 1 | sim_small |
mock:small |
0.853 | 83.3% | $0.06 | 190ms | ranked |
| 2 | sim_frontier |
mock:frontier |
0.750 | 97.6% | $0.30 | 1,500ms | ranked |
| 3 | sim_balanced |
mock:balanced |
0.744 | 85.7% | $0.18 | 620ms | ranked |
| — | sim_tiny |
mock:tiny |
— | 50.0% | $0.02 | 90ms | DISQUALIFIED |
Normalised contributions
Each metric scaled to 0–1 (1 = best in this field), multiplied by its weight.
| model | accuracy ×0.55 | cost ×0.25 | latency ×0.20 | composite |
|---|---|---|---|---|
sim_small |
0.83 → 0.458 | 0.97 → 0.243 | 0.76 → 0.152 | 0.853 |
sim_frontier |
0.98 → 0.537 | 0.85 → 0.213 | 0.00 → 0.000 | 0.750 |
sim_balanced |
0.86 → 0.471 | 0.91 → 0.228 | 0.22 → 0.045 | 0.744 |
Disqualified
| model | reason |
|---|---|
sim_tiny |
accuracy 50.0% below the required 70.0% |
Accuracy by tag
| model | account | ambiguous | billing | easy | hard | refund | shipping | spam | technical |
|---|---|---|---|---|---|---|---|---|---|
sim_small |
83% | 67% | 100% | 90% | 80% | 83% | 83% | 100% | 78% |
sim_frontier |
100% | 100% | 83% | 100% | 93% | 100% | 100% | 100% | 100% |
sim_balanced |
100% | 100% | 67% | 90% | 80% | 100% | 83% | 100% | 78% |
sim_tiny |
50% | 67% | 67% | 48% | 60% | 83% | 50% | 33% | 33% |
Per-test results
| test | eval | sim_small | sim_frontier | sim_balanced | sim_tiny |
|---|---|---|---|---|---|
double_charge |
classification | ✓ | ✓ | 0.67 | 0.33 |
card_declined |
classification | ✓ | 0.67 | 0.67 | ✓ |
app_crash |
classification | ✓ | ✓ | 0.67 | 0.67 |
api_500 |
classification | 0.67 | ✓ | ✓ | 0.33 |
sso_loop |
classification | 0.67 | ✓ | 0.67 | ✗ |
password_reset |
classification | 0.67 | ✓ | ✓ | 0.67 |
seat_transfer |
classification | ✓ | ✓ | ✓ | 0.33 |
parcel_late |
classification | ✓ | ✓ | ✓ | ✗ |
address_change |
classification | 0.67 | ✓ | 0.67 | ✓ |
money_back |
classification | ✓ | ✓ | ✓ | ✓ |
refund_vs_billing |
classification | 0.67 | ✓ | ✓ | 0.67 |
crypto_spam |
classification | ✓ | ✓ | ✓ | 0.33 |
Where models went wrong
| model | test | output | expected | why it failed |
|---|---|---|---|---|
sim_balanced |
double_charge |
I am not sure. | billing | output named none of the labels ['billing', 'technical', 'ac… |
sim_balanced |
sso_loop |
I am not sure. | technical | output named none of the labels ['billing', 'technical', 'ac… |
sim_small |
password_reset |
I am not sure. | account | output named none of the labels ['billing', 'technical', 'ac… |
sim_small |
refund_vs_billing |
I am not sure. | ['refund', 'billing'] | output named none of the labels ['billing', 'technical', 'ac… |
sim_tiny |
app_crash |
I am not sure. | technical | output named none of the labels ['billing', 'technical', 'ac… |
sim_tiny |
crypto_spam |
I am not sure. | spam | output named none of the labels ['billing', 'technical', 'ac… |
sim_tiny |
parcel_late |
I am not sure. | shipping | output named none of the labels ['billing', 'technical', 'ac… |
sim_tiny |
password_reset |
I am not sure. | account | output named none of the labels ['billing', 'technical', 'ac… |
2 more failure(s) in the database.
Reproduce
arena evaluate --project projects/support_triage
Raw results: projects/support_triage/results/arena.sqlite (table results, run id run_20260806_174425_c8a317).
Source: docs/EXAMPLE_REPORT.md