NEEDS_HUMAN is a first-class terminal status, not an exception — which is what makes
escalation architectural rather than bolted on.
The operator works the same live session, guarded by a Lease{holder, epoch}. Automation
checks it on every dispatch — who holds the session, and has it changed hands since this run was
authorized — either returns LEASE_LOST; both matter, since the epoch alone once left a replay
running to SUCCESS while an operator held the session, because the check was opt-in and unasked.
Human input does not bypass policy. The console never injects events into the page; it submits
raw_input to the broker, through the same resolve → authorize → dispatch → evidence path under a
HUMAN profile — escalation widens authority deliberately and auditably, and the allowlist and
redaction apply unchanged.
Control comes back. ReplayExecutor.resume() re-anchors against the live screen rather than
run() with an offset: it skips the entrypoint navigation (which would reload the frameset and
discard the operator's state), re-adopts the epoch the handoff produced, carries forward outputs
and spent recovery attempts, and fails closed when the screen matches no step. A step is skipped
only when its postcondition holds — without one, resume would re-submit the search the operator
just ran. cua console --capability <ref> --arm-fault <fault> runs a capability against the
supervised session, so the queue a reviewer opens has something waiting in it.
One chokepoint, three enforcements (§1), applied identically to all three actors.
Allowlist — domains, URL patterns, action types, step and duration caps, in
config/policy.yaml. Navigation outside it is NAVIGATION_BLOCKED, for
humans too; a capability's own allowed_domains only ever narrows the global list, checked both
on a navigate's declared URL and again against where the session actually landed, since checking
intent is not checking outcome. "Landed" means every frame, not the top-level document — on a
frameset the shell never moves — and the driver also refuses an off-list document navigation in any
frame, redirect hops included, before it is fetched.
Risk tiers — safe → elevated → irreversible, from three independent signals (action type ×
target semantics × artifact annotation), since any one alone is fooled. Automation is refused an
irreversible action unless the step is declared irreversible and the run carries an approval of
that exact content (or of its sealed base, through a --base-url-only binding) and a caller
opt-in — checked by PolicyEngine at the dispatch itself, not only before the run, because either
gate alone is one accident away from a wire transfer.
Redaction at every sink, including outbound model prompts. Writing a run record requires a
redactor, and the secret resolver registers each value as it issues it, closing a path where an
unregistered secret ran the redaction pass empty. Tested in both directions — id@version once
matched the email pattern, so a run record named its own capability <redacted:email>. The
operator's live view is redacted from the same sensitive declarations an evidence screenshot is,
being the sink most likely to sit on a second monitor. make secrets scans every commit in CI.
Limits, stated rather than claimed away. Free-text redaction is patterns plus registered
literals, so a member name is protected only where a capability declares that field sensitive,
not by shape. Prompt injection is scoped, not solved: a closed action space and policy evaluated
outside the model, but a model talked into a permitted action on a permitted target still
performs it. Fixture sign-in (--sign-in) drives the browser directly — the one documented
carve-out, opt-in and scoped to the mock app's fixed credentials. No real credentials or PII exist
in this repository.
Scope the brief did not ask for. §3.6 scopes a co-browsing console out and invites a mock; I built a working one, because control transfer was the part I most wanted to prove end to end — ~5,000 lines against the brief's own preference for "small, correct, well-argued", and the first thing I would cut. Not built: continuous pixel streaming (a still frame on connect and after each policed gesture, not co-browsing); a real desktop surface beyond the interface-only UIA stub; multi-operator routing, SSO, audit sign-off.
Stretch goals: one taken seriously. §8 invites at most one or two; I took cross-tenant reuse
(§4) — the hardest to fake, and the only one testing whether the artifact is portable or merely a
recording. The rest fell out of making that claim checkable: a catalog and tool schema (§2) to
invoke a capability by name, cua eval to measure portability, cua catalog approve to gate what
is not yet measured (refusing evidence that does not support it, losing it the moment the artifact
is edited), and cua codegen, emitting a runnable Playwright test as the same argument aimed
outward — CI runs the generated file rather than trusting it. Assisted recovery
(cua replay --assist) is the one I would have left out: one model call after a deterministic run
has failed, dispatched through the chokepoint from cua.assist, above both cua.agent and
cua.replay so it cannot make §3's central claim false. A plain replay still makes zero model
calls, asserted rather than described.
What the evidence shows.
evidence/discovery/disc-e0aa86b951/ is a genuine
LLM-driven run: six model calls, 10,215 tokens, each carrying the gateway's own provider and
request id (make verify-live re-checks any run; this one predates latency tracing, so reconcile
its request ids with the gateway's logs). The artifact compiled from it,
memberdesk.savings_balance@1.0.0, replays at zero drift for members
that run never saw.
Known weaknesses, stated rather than hidden. The compiler reads a value by the label beside it
— right on a label/value table, wrong in an n-column grid, where it cannot yet read a column
header and falls back to a flagged positional descriptor. raw_input is classified by what it
lands on — a click is hit-tested against the controls' geometry, a key against the focused control
— so a click on "Transfer Funds" needs the operator's explicit confirmation; a gesture on something
the semantic tree does not expose falls back to its action type's tier. Approval pins the sealed base; a --base-url binding is re-derived at the gate and
covered only because it fills a declared slot — a tenant overlay that overrides steps has no approval
path of its own yet, so an irreversible capability behind one fails closed.
Two failures of the same shape: a check existed, the test was green, and nothing was actually
checked. make invariants ran import-linter through python -m importlinter.cli — not a
runnable entry point, so every "mechanically enforced" claim here held only once that became
lint-imports. And the compiler wrote the discovery host into entrypoint.url_pattern verbatim,
so --base-url substituted nothing, working only on the recording machine — now parameterised at
compile time, and pinned under a test.
Then the question was whether the suite could fail at all. A hardening pass wrote about 1,800
edge-case tests — boundary, malformed, hostile and out-of-order input for every subsystem, the
command line included — and the defects they exposed were real, and are all fixed: an allowlist that
raised on a malformed URL instead of denying it; the headline wrong-action metric ignoring the
expected status; a sealed artifact that reloaded as tampered after declaring an empty list; an
identifier that was all digits one time in 300 and was redacted as an account number in its own
trace; one redaction pattern that took minutes on a single long token. make mutation-check then
breaks the code on purpose — each of the ten invariants, and a guard per fix — and records whether
the suite noticed: 22 of 22 deliberate violations are caught. make edge-cases keeps 22 scenarios
as saved runs, so the boundary conditions can be opened and read rather than only asserted.
| The artifact | evidence/capabilities/ |
| The handoff, actors and epochs | evidence/escalation/demo-escalation/trace.jsonl |
| All four terminal statuses | evidence/replay/, evidence/escalation/ |
| The invariants, enforced | .importlinter, tests/invariants/ |
| The whole story, one command | make demo |
| The numbers behind §3 and §4 | evidence/evals/ |
| The edge cases, run and saved | evidence/edge-cases/ |
| Does the suite fail when it should? | evidence/tests/mutation-report.md |
| One page of results | evidence/SUMMARY.md |