Testing and evaluation

Two different questions, deliberately kept apart:

• Tests ask does it work — deterministic, free, and on every commit.
• Evaluations ask is the output any good — real models, real money, run on purpose.

The pyramid

LLM_MODE=fake is what makes the middle layers deterministic and free: the factory returns scripted models and the retriever chain returns fixture results, so no network call and no API key is involved.

The golden journeys

Five journeys run in CI against the packaged stack, and they encode the product's promises.

Journey 2 is the load-bearing one: it asserts that resume enters the graph at the gate, not at the planner.

cd frontend && npm run e2e

Running it locally. The suite drives the real stack, so bring one up first — ./start.sh --fake is enough, and fake mode is what CI uses. Then point Playwright at it:

E2E_NO_SERVER=1 E2E_BASE_URL=http://localhost:3031 npm run e2e

Omit E2E_NO_SERVER and Playwright builds and starts the frontend itself; set it when one is already running, which is faster and what CI does.

Repeated local runs are safe, and that is not an accident. Registration is capped at 5 per IP per hour and the cap is not configurable — it is brute-force protection, and one a test suite can switch off is not a cap. The suite registers one account per journey and has more journeys than that, so it would exhaust the cap on the first run and fail on the second. e2e/fixtures.ts therefore deletes this suite's own rl:* counters before each journey; it never issues FLUSHDB, because the same Redis carries the SSE channels and the search cache and blowing those away would change what the run under test actually does.

If you do hit the cap — running one spec against a stack whose Redis the fixture cannot reach, say — the failure now names itself rather than surfacing as a navigation timeout. Clear the counters by hand with:

redis-cli -n 1 --scan --pattern 'rl:*' | xargs -r redis-cli -n 1 del

The cap itself stays armed: backend/tests/security/test_auth_rate_limit.py proves a sixth registration from one address is refused, and pins the published limit so widening it has to be deliberate.

CI starts the API and the worker in fake mode, then Playwright. It waits for an explicit worker preload signal rather than for a ping — a Celery worker answers a ping while LangGraph and the provider clients are still importing, which measurably cost tens of seconds on the first task and made these journeys look flaky when the real cause was a worker that had not finished booting.

Testing an absence

A recurring trap worth its own section. When you write a test for an absence — no network egress, no writes, no spend — check what the fixtures replaced.

A test once asserted zero network calls in corpus mode while injecting a fake embedder, and the query embedding was the only call that egressed. The suite was green precisely because it had replaced the defect. If the mechanism under test is the thing you mocked, the test is decorative.

What CI enforces

Run what CI runs, not what is convenient:

cd backend && ruff check app/ research_engine/ tests/ evals/ \
  && ruff format --check app/ research_engine/ tests/ evals/ \
  && python -m pytest

cd frontend && npm run lint && npm run typecheck && npm test && npm run build

Plus, in the frontend job, four bespoke greps that npm run lint does not run.

Jobs: backend, frontend, golden-e2e, and eval-artifacts. Dependency audits (pip-audit, npm audit) run non-blocking.

Evaluation results are write-once

The eval-artifacts job fails the build if a committed file under backend/evals/results/ is modified. Adding a new result is always fine; changing one is not.

This exists because a frontend commit once silently overwrote a real 10/10 measurement with a failed run, leaving the README citing numbers whose proof no longer existed. Nothing failed and nothing warned. Record a new run under a new name — eval-<date>-<routing>-run<N>.json, because a date alone is not a run identity.

Evaluations

The harness runs a versioned query set through the compiled graph up to the review gate and records per-report metrics to a dated JSON file, so report quality is diffable over time.

make eval                                   # fake mode: deterministic, free, no keys
LLM_MODE=real make eval EVAL_ARGS="--judge anthropic:claude-sonnet-4-6"   # real models

Mode is read from the real environment before .env is loaded, and .env supplies keys only. That ordering is deliberate: a developer .env commonly carries LLM_MODE=real for the app, and letting that reach the harness would silently turn make eval — documented and relied on as the free default — into a run that spends money on every invocation. Spending is opt-in, per invocation.

Metrics

Every run records its own method block and a metrics_version, bumped whenever a definition changes, so two runs are never silently compared across incompatible metrics.

Retrieval quality

Report quality is bounded by retrieval quality — docs/01, docs/10, docs/21 and docs/33 all say so — and for a long time nothing measured it. A second harness does now, separate from the report harness above because it answers a narrower question: given a query, does the retriever return the right documents?

python -m evals.retrieval             # measure and print
python -m evals.retrieval --write     # also record a write-once result file

It measures; it does not rank. Queries go through CorpusStore.search — the same call the executor makes — so what is recorded is production behaviour rather than a reimplementation of it. The dataset is a frozen 16-document corpus with a fingerprint, so a changed corpus cannot be mistaken for a changed retriever.

A real embedder is required, and its absence is reported rather than filled in. FakeEmbeddings hashes text with blake2b — deterministic, and carrying no semantic signal — so recall over those vectors would be a number with no meaning attached. With no embedding endpoint reachable the harness refuses and records "status": "unmeasured" with the reason. That is the same rule as Unmeasured is not zero, one level up.

The recorded baseline (backend/evals/results/retrieval-2026-09-09-corpus-local.json, ollama:nomic-embed-text, 12 queries):

This baseline is the gate a retrieval change has to clear: no change merges without a measured, committed result either side of it. Hybrid retrieval was built and measured against exactly this and did not beat it, so it was rejected — see the roadmap.

Retrieval over selected files

python -m evals.selection_retrieval           # measure and print
python -m evals.selection_retrieval --write   # also record a write-once result file

A run over selected files searches only those files (ScopedCorpus.search). This eval checks two things about that path, on the same embedder:

• Leakage. The share of retrieved chunks from a file outside the selection. The target is exactly 0, and the command exits non-zero when any query leaks. The dataset (evals/datasets/selection-retrieval/) is built so a leak has something to find: one fixture of every shipped format plus the corpus above, where unselected files often answer the query better, and two selections leave out an identical copy of the chosen file.
• The per-file hit cap. A selection takes max(1, ceil(k / files)) hits from each file, where every search used to take one. It is measured against one hit per file. Document rankings are identical by construction, so the measure is passage recall: whether every passage a question needs comes back, judged by verbatim anchors.

The recorded result (backend/evals/results/retrieval-2026-10-28-selection-local.json, ollama:nomic-embed-text, 16 queries, 35 documents):

Unmeasured is not zero

The rule the whole harness is built around, and the reason it exists at all.

A provider error, a partial model reply, and an unparseable answer must not each silently contribute a miss to a quality score — that makes an exhausted quota indistinguishable from a quality collapse. So:

• unjudged claims are excluded from the denominator, not counted as failures;
• a metric returns None when nothing could be judged, and None renders as n/a (unmeasured), never 0.0;
• a rate over nothing is None too — a completion or pass rate over zero queries, and an average over none. 0.0 completion would read as "every query failed" and report a failure against the threshold that never happened;
• a threshold check has three answers: met, missed, or None when there was nothing to measure. None is never reported as a miss;
• a trace records the model that actually answered, never the one that was requested.

The result file stores an unmeasured value as null; the console prints it as n/a (unmeasured), the same words the harness, the benchmark and the retrieval eval all use.

The rule has two homes — the harness and the benchmark — and they must change together.

The judge

Citation support is graded by the model --judge names, and a real-mode report eval refuses to start without one. It is refused, before a single query runs, if it is:

• a route the pipeline itself runs, or the same model reached another way (openrouter:google/gemini-2.5-flash is google:gemini-2.5-flash);
• a router alias (auto/*), which can resolve to a different model on every call;
• set while any pipeline role is on a router alias, since then nothing can be shown to be independent of it.

Judging never falls back to the pipeline's own critic. That was the harness's behaviour until V3, which is why the runs recorded before it are self-judged.

The result file names the judge in a judge block beside models, never inside it: models is the system under evaluation, and judge is what graded it. The block holds:

• the judge's route;
• the routes it was checked against;
• for the baseline and any candidate, how many claims it ruled on, how many it never reached, and which model the provider reported answering;
• whether it answered at all.

Each claim verdict also records the model that served it. That name is read from the provider's response, never copied from the route: a judge that answered as something else, or did not disclose what answered, shows it. A scripted run calls no judge and writes no judge block.

The release run

RG-5 is this harness on the fixed query set with a named judge and the committed candidate, nothing more. The pipeline runs one research task at a time: a hosted route that throttles bursts (Antigravity via OmniRoute returned empty responses after a burst of parallel calls in the preflight) would otherwise fail queries for reasons that have nothing to do with quality.

cd backend
LLM_MODE=real MAX_PARALLEL_TASKS=1 \
  MODEL_PLANNER=custom:antigravity/gemini-2.5-flash MODEL_EXECUTOR=custom:antigravity/gemini-2.5-flash \
  MODEL_CRITIC=custom:antigravity/gemini-2.5-flash MODEL_SYNTHESIZER=custom:antigravity/gemini-2.5-flash \
  MODEL_CHAT=custom:antigravity/gemini-2.5-flash \
  python -m evals.harness --judge custom:kiro/claude-sonnet-4.5 \
    --candidate evals/candidates/v3-release-planner.json

Parallelism changes wall-clock, not what is measured, and the result records it: method.limits holds every execution limit the run used, so a result can be reproduced from its own file.

Release criteria

Citation support ≥ 0.95 and completion ≥ 0.90 on the fixed set. Both are inclusive — a run at exactly the threshold meets it — and both are fixed: nothing a user configures changes them. The V3.0.0 release run clears both under an independent judge; see Citation-fidelity benchmark for what has actually been measured, and under what caveats.

Custom-spec evaluation

A user can replace a role's system prompt, so the harness can measure a replacement against the shipped prompts:

cd backend && python -m evals.harness --candidate spec.json   # {"planner": "…", "critic": "…"}

It runs the fixed ten-query set twice — once with the shipped prompts, once with the candidate — and judges both against the same thresholds. The result file keeps the shipped run where a plain run puts it (aggregate, release_criteria, results) and adds the candidate beside it under candidate, with each replaced prompt and its SHA-256. It is written to its own write-once file, eval-<date>-<routing>-custom.json, so it can never land on a plain run's result.

A candidate below threshold is reported, not blocked. The user chose it; the harness's job is to measure honestly, not to refuse. A threshold miss never stops the process — the only exit is the write-once guard.

The candidate is refused, not quietly substituted, if it cannot be used. It is validated by the same all-or-nothing rule a run's own override snapshot goes through. In production an unusable snapshot falls back to the shipped prompts and records that it did; here the same fallback would publish the baseline's numbers under the candidate's name. It comes from the file you pass, never from a user's stored preferences, and applies to the report eval only.

What the candidate can and cannot reach

A user's prompt may change what the pipeline produces, never what the harness measures. Two independent protections keep the judge out of reach:

• Scope. The candidate is installed around the pipeline run alone and removed before the judge is called. It is never installed process-wide, which would outlive the query and still be live when the judge ran. Both runs are built from the same base configuration, so they differ in their prompts and nothing else — the same models, the same limits.
• Construction. Both judges carry their system prompt as literal text and never compose one through the prompt machinery. A replaced prompt has no path into a judge's messages even if it were live at the moment the judge ran.

The project-chat prompt the memory eval composes is a protected purpose, so no override reaches it either.

What scripted mode proves, and what it does not

make eval runs scripted models, and so does CI. That proves the mechanism — the candidate reaches the pipeline, both runs are recorded, the same thresholds apply, the judge never sees the candidate. It does not show that one prompt is better than another: the judge runs only against a real provider, so citation support is unmeasured for both runs, and the scripted models answer by role, so a candidate produces the baseline's output. Comparing the quality of two specs needs a real-model run.

Quality gates