DOCS
Documentation
Documentation for the Computer-Use Automation system. Every statement describes what is built, or is marked explicitly as planned. New here? Start with Quick Start, then Demo Walkthrough.
Getting Started
What this is, how to run it, and how to configure a model — including a local one.
- Quick Startsetup + dependencies
- Demo Walkthrough14 stages, end to end
- CLI Referenceevery command
Architecture
How the system is put together, and why the boundaries fall where they do.
- System Architecturepackage map + dependency rules
- Execution PipelineAction → Resolver → Policy → Driver
- Ten Invariantsmechanically enforced contracts
Design
Detailed design documents — the artifact schema, target resolution, error taxonomy, and human-in-the-loop control transfer.
- Capability Artifactdesign/artifact-schema
- Target Resolutiondesign/target-resolution
- Error Taxonomydesign/error-taxonomy
- Control Transfer (HITL)design/control-transfer
Decisions (ADRs)
Why it is this way — the architectural decision records.
- ADR 0001 — UiSnapshot as cross-surface abstractionaccepted
- ADR 0002 — Immutable content-addressed capabilityaccepted
- ADR 0003 — Four-class observation taxonomyaccepted
- ADR 0004 — Single policy chokepointaccepted
- ADR 0005 — Tenant overlay, not forkaccepted
Product
- Requirements Traceabilityrequirements-traceability
Testing
- Testing Strategyclaim → test table
Runbooks
Step-by-step operating procedures.
- Run a Discoveryrunbooks/run-a-discovery
- Run a Replayrunbooks/run-a-replay
- Handle an Escalationrunbooks/handle-an-escalation
Quick Start
Get the system running in under a minute.
Prerequisites
- Python 3.11+
uv (optional, used if present; otherwise venv + pip)
Setup
git clone https://github.com/adityamhaske/computer-use-automation.git
cd computer-use-automation
make setup
Configuration
cp .env.example .env
The only thing that needs an API key is the live discovery run. Replay, escalation, the full test suite, and the demo all work without one.
| Variable | Purpose |
OMNIROUTE_API_KEY | Discovery only. Any OpenAI-compatible gateway works; OPENROUTER_API_KEY is still read as a fallback. Pair with CUA_LLM_BASE_URL and CUA_LLM_MODEL. |
CUA_LLM_BASE_URL | The gateway's /v1 endpoint (OmniRoute). Blank falls back to OpenRouter. |
CUA_LLM_MODEL | Model ID for discovery. Name a concrete model: a router alias such as auto hides which one ran. |
CUA_POLICY_FILE | Allowlist and risk policy — defaults to config/policy.yaml |
Run the demo
make demo
Takes about a minute. Works without an API key — discovery falls back to a recorded transcript and says so explicitly.
Unified Orchestrator (./start.sh)
Orchestration over the Makefile targets — it adds nothing you could not run by hand, it just
names the combinations (app and console together, wait-for-ready, teardown on Ctrl+C).
./start.sh
./start.sh ui
./start.sh ui --headed
./start.sh ui --bare
./start.sh app
./start.sh app-b
./start.sh check
./start.sh test
./start.sh eval
Development commands (Makefile)
make check
make test
make invariants
make secrets
make discover
make verify-live
make eval
make edge-cases
make test-report
make mutation-check
make evidence
2,158 tests, mypy --strict clean, five enforced import contracts. The whole suite runs offline with no API key. make test-report writes the counts and coverage to evidence/tests/; make mutation-check breaks each invariant on purpose and proves the suite goes red.
Demo Walkthrough
make demo boots the mock back-office and runs fourteen stages end to end, writing everything to evidence/.
| # | Stage | What happens | Result |
| 1 | Mock back-office running | Hostile frameset app, no test IDs | READY |
| 2 | Discovery | The model drives the live UI to the goal | TRACE |
| 3 | Capability compiled and sealed | Typed I/O, semantic targets, content hash | SEALED |
| 4 | Deterministic replay, new inputs | No model in the loop | SUCCESS |
| 5 | Business outcome | member_not_found — an answer, not a crash | EXIT 0 |
| 6 | Injected 502 | Declared recovery cleared it | SUCCESS |
| 7 | Fault past budget | Recovery exhausted → escalate | NEEDS_HUMAN |
| 8 | Malformed input | Rejected before acting | FAILED |
| 9 | Undeclared screen | Failed closed and escalated | NEEDS_HUMAN |
| 10 | Operator took over | Drove the same live session, handed it back | EPOCH 3 |
| 11 | Re-anchored after handoff | Resumes at the right step, not step 1 | SUCCESS |
| 12 | Agent invocation | Called by name with typed args and declared outcomes | SUCCESS |
No API key required. Stage 2 falls back to a recorded transcript and says so, in the output and in the evidence. Everything after it is model-free by construction. Set OMNIROUTE_API_KEY for a genuine LLM-driven discovery, then make verify-live to check the run was live.
Fault injection
Faults are injected into the mock app, not the replay engine — the executor must not know a fault is coming:
curl -X POST localhost:8811/_control/arm \
-d '{"fault":"transient_load","count":1}' \
-H 'content-type: application/json'
curl -X POST localhost:8811/_control/reset
Available faults: transient_load, session_timeout, undeclared_dialog, validation_error.
Exit codes
| Status | Exit | Meaning |
SUCCESS | 0 | Checkpoint verified, typed outputs returned |
BUSINESS_OUTCOME | 0 | A legitimate answer — "no such member" is not a failure |
NEEDS_HUMAN | 2 | Stopped safely; an intervention is queued |
FAILED | 1 | A defect. Carries step, expected, observed. |
CLI Reference
make demo is orchestration, not a second implementation — every stage is a command you can run on its own.
Discovery
cua discover \
--goal "Look up member 12345 and read their current savings balance" \
--target http://localhost:8811 \
--out evidence/capabilities/
Requires OMNIROUTE_API_KEY. The model drives a live UI, the run is compiled into a capability artifact.
Replay
cua replay <artifact> --base-url http://localhost:8811 --input member_id=67890
cua replay <artifact> --base-url http://localhost:8811 --input member_id=99999
Operator Console
cua console --target http://localhost:8811
Catalog
cua catalog list
cua catalog show <id> --tool-schema
cua catalog invoke <id> --input member_id=12345 --base-url http://localhost:8811
Agent Demo
cua agent-demo --base-url http://localhost:8811
Mock App
make app
make app-variant-b
System Architecture
How the pieces fit together, and why the boundaries fall where they do.
Package map
| Package | Owns | May import |
cua/domain | Types: Capability, TargetDescriptor, Action, UiSnapshot, RunRecord, taxonomy | nothing from cua |
cua/perception | Normalizing raw surface observation into UiSnapshot; fingerprinting | domain |
cua/targeting | Resolution ladder, scoring, ambiguity, drift | domain, perception |
cua/policy | Allowlist, risk tiers, redaction, secrets, AuthorizedAction | domain, targeting |
cua/hitl | Lease, SessionBroker, intervention queue, operator console | domain, policy |
cua/runtime | dispatcher.py — the sole importer of surfaces | everything below |
cua/surfaces | SurfaceDriver port; Playwright/CDP driver; desktop stub | domain, perception |
cua/agent | LlmPort, prompts, the discovery loop | runtime and below |
cua/replay | The deterministic executor | runtime and below — never agent |
cua/recorder | Compiling a discovery trace into a Capability | domain, targeting |
cua/evidence | Structured, redacted, actor-tagged run records | domain, policy |
cua/catalog | Capability store, tenant overlays, tool-spec export | domain |
cua/cli | Entry points | everything |
Project layout
src/cua/ domain · perception · surfaces · targeting · policy · runtime
agent · recorder · replay · hitl · evidence · cli
apps/mock_bank/ the hostile target application (+ a "second tenant" variant)
docs/ ADRs (why) · design (how) · prd (what, in what order) · runbooks
tests/ unit · integration · contract · invariants · e2e(live)
evidence/ proof that the end-to-end thread actually ran
including evals/ — the measured numbers behind the claims
Execution Pipeline
One path to a surface, for every actor. Discovery, replay, and human intervention all take the same path.
Action ──► TargetResolver ──► PolicyEngine ──► SurfaceDriver
(resolve or refuse) (authorize) (dispatch)
Enforced three independent ways:
- Import rule.
cua.runtime.dispatcher is the only module allowed to import cua.surfaces (.importlinter contract policy-chokepoint).
- Type rule.
SurfaceDriver.dispatch() accepts only an AuthorizedAction, which carries a token instance held exclusively by PolicyEngine.
- Reconciliation test. Every
dispatch event in a run record must have a matching authorize event with the same action ID.
Human input is policed, not injected
The console submits raw_input actions which traverse the same path under a distinct HUMAN policy profile:
| AUTOMATION | HUMAN (during intervention) |
| Domain allowlist | enforced | enforced |
safe actions | allow | allow |
elevated actions | allow if declared elevated by the step | allow, recorded |
irreversible actions | refused, unless the step is declared irreversible and the run carries an approval of that exact content plus the caller's opt-in (allow_if_approved, checked at dispatch) | allow with confirmation, recorded |
| Off-artifact navigation | NAVIGATION_BLOCKED | allowed within allowlist, recorded |
| Evidence redaction | applied | applied |
The Ten Invariants
The value of this system is almost entirely in a handful of properties. Each is mechanically enforced — the test is named so you can see it fail on purpose.
1. Replay never calls or imports the LLM
cua.replay may not import cua.agent, httpx, or any model client — directly or transitively. A replay that can quietly ask a model for help is not deterministic, not cheap, and not auditable.
2. No action reaches a driver without policy authorization
cua.runtime.dispatcher is the only module permitted to call dispatch(). SurfaceDriver.dispatch() accepts only an AuthorizedAction, which carries a token that only PolicyEngine can mint.
3. Target identity is a semantic TargetDescriptor, never a raw CSS selector
Artifacts describe controls by role, accessible name, scope, structural anchor, and ordinal. Surface-specific data may appear only inside TargetDescriptor.hints, and a hint is an unverified cache.
4. Human input cannot bypass policy or evidence
The operator console never injects into the page. It submits raw_input actions to the SessionBroker, which runs them through the same chokepoint under the HUMAN policy profile.
5. Unknown states fail closed
If the observed UI matches no declared precondition, checkpoint, outcome, or recovery rule, it is UNEXPECTED_STATE and the run stops. If a target resolves ambiguously, the resolver refuses with TARGET_AMBIGUOUS.
6. Every sink is redacted
Logs, artifacts, evidence files, screenshots, and outbound LLM prompts. Secrets are {$secret: ref} references resolved at dispatch time and never written anywhere.
7. Business outcomes are not failures
"No such member" is a legitimate answer the caller needs, not a crash. BUSINESS_OUTCOME exits 0.
8. Automation may only act while it holds the session
Every dispatch asks the lease: who holds this session, and has it changed hands since this run was authorized. Either answer can refuse the dispatch with LEASE_LOST.
9. Capabilities are immutable
A Capability at id@version is frozen and content-hashed. Anything that changes because you ran it lives in a separate document: RunRecord, CapabilityEvaluation, CapabilityApproval.
10. domain/ is pure
No I/O, no network, no browser, no clock. cua.domain imports nothing else from cua.
The Capability Artifact
The unit of reuse in this system. What the LLM's discovery run compiles down to, what a human reviews, and what an AI agent invokes in production.
Decision rationale lives in ADR 0002. This document is the reference for what the schema is.
Three readers, one artifact
| Reader | Needs |
| Replay engine | Unambiguous, executable steps with explicit success and failure conditions |
| Human reviewer | To understand what it does, what it touches, and what could go wrong — without running it |
| Calling AI agent | A typed contract: what to pass, what comes back, what can happen |
Document family
Only the first is the capability. The rest exist so the capability can stay immutable.
cua.capability/v1 frozen, content-hashed — the contract
cua.tenant_binding/v1 per-tenant vars + step overrides (base ⊕ overlay)
cua.run_record/v1 one execution: trace, resolutions, drift, evidence refs
cua.capability_evaluation/v1 aggregate over runs: stability, strategy mix, drift trend
cua.capability_approval/v1 draft → approved, approver, scope
Identity
capability:
id: "corebank.member.savings_balance"
version: "1.2.0"
content_hash: "sha256:…"
id is namespaced by vendor product, never tenant — the precondition for cross-tenant reuse.
Typed contract
inputs:
- { name: member_id, type: string, required: true, pattern: "^[0-9]{4,10}$",
sensitive: false, example: "12345" }
outputs:
- { name: savings_balance, type: money, source: { step: read_balance } }
These emit JSON Schema directly, so the artifact is the agent tool contract — there is no second source of truth to drift. sensitive: true propagates everywhere.
Steps
steps:
- id: enter_member_id
action: { type: type, value: { $input: member_id } }
target:
role: textbox
name: { match: exact, value: "Member Number" }
scope: { frame: content, region: search_form }
anchor: { relation: row_of, cell_text: "Member Number" }
hints: { css: "...", node_path: "..." }
fingerprint: "sha256:…"
recorded_strategy: semantic_exact
precondition: { assert: node_exists, role: textbox, name: "Member Number" }
wait: { for: snapshot_stable, timeout_ms: 5000 }
postcondition: { assert: node_has_value, value: { $input: member_id } }
risk: safe
Every step asserts both before and after. A step that only acts is a step that assumes the click worked.
Success, outcomes, recovery
checkpoint:
all_of:
- { assert: node_exists, role: heading, name_contains: "Member Detail" }
- { assert: node_matches, role: cell, name_matches: '^\\$[0-9,]+\\.[0-9]{2}$' }
outcomes:
- { code: member_not_found, detect: { text_present: "No records found" },
terminal: true, returns: { member_id: { $input: member_id } } }
recovery:
- { id: transient_load, detect: { http_status_in: [502,503,504] },
remedy: [ { reload: {} } ], max_attempts: 3, backoff: exponential }
Target Resolution
Resolution is where determinism is won or lost. Turning a semantic TargetDescriptor into a concrete UiNode in the current snapshot.
The resolution ladder
Strategies are tried in fixed order. The first rung that yields any candidate decides: exactly one wins; more than one is refused unless the descriptor carries an explicit ordinal. A rung with several candidates does not fall through to the next.
| # | Strategy | Matches on | Survives |
| 1 | semantic_exact | role + accessible name, exact, unique in scope | most things |
| 2 | semantic_normalized | same, case/whitespace/punctuation folded | cosmetic label edits |
| 3 | structural_anchor | position relative to nearby text | controls with no usable name at all |
| 4 | hint_cached | stored CSS/node-path hint | nothing structural; speed optimization |
| 5 | ordinal_in_region | Nth control of a role within a scope | rebranding; last resort before refusal |
| 6 | vision | OCR / bounding box | not implemented — no generator exists; replay excludes it by construction |
Rung 3 is what makes legacy enterprise apps tractable. Those apps are full of controls with no name, no label, and no test ID, sitting in a table cell next to the text that identifies them.
Rung 4 is a cache, not an identity. A hint is accepted only if the node it resolves to also satisfies the semantic assertion.
Ambiguity is refusal
If a rung yields two or more candidates and the descriptor carries no explicit ordinal, the resolver raises TARGET_AMBIGUOUS and the run stops. It does not pick the first, the topmost, or the highest-scoring. There is no threshold to tune: the reported ambiguity figure, (candidates − 1) / candidates, is for debugging and is never compared against anything.
Two plausible "Submit" buttons mean the screen is not what we think it is, and acting on either is how an automation posts a transaction to the wrong account. A refusal is recoverable — a wrong click may not be.
Drift
Resolving via a lower-priority strategy than the one recorded at discovery is drift — a signal, not a silent success. Per-step drift aggregates into a run-level drift_score.
This is the mechanism behind the multi-tenant story: a tenant whose UI has diverged shows rising drift before replays begin to fail.
Variant B: two kinds of divergence
Case A — markup churn, same vocabulary (the ladder handles this)
| Base | Variant B |
| accessible name | Member Number | Member Number — same |
| form field name | memno | member_num — changed |
| CSS class | frmfld | ng-inp — changed |
semantic_exact resolves it on both. This is the falsifiable form of "this system does not depend on CSS selectors."
Case B — rebranding, different vocabulary (this needs an overlay)
| Base | Variant B |
| balance label | Savings Balance | Savings Bal. |
| account type | Account Type | Type of Account |
No automatic recovery here — and that is correct. Inferring that "Savings Bal." means "Savings Balance" is a fuzzy guess, and a system that guesses which row holds a balance will eventually read the wrong one.
Error Taxonomy
Two distinct type families — what we observed, and how the run ended — because they are not the same axis.
What we observed (a classification of a moment)
ObservationClass = EXPECTED | BUSINESS_OUTCOME | RECOVERABLE | HARD_FAILURE | UNEXPECTED_STATE
How the run ended (a terminal status)
RunStatus = SUCCESS | BUSINESS_OUTCOME | NEEDS_HUMAN | FAILED
The decision procedure
At every observation point the executor asks, in this order:
1. Does the state satisfy the step precondition / checkpoint? → EXPECTED, proceed
2. Does it match a declared `outcomes` detector? → BUSINESS_OUTCOME, return
3. Does it match a declared `recovery` detector? → RECOVERABLE, remediate
4. Is it a known hard-failure condition? → HARD_FAILURE, stop
5. Otherwise → UNEXPECTED_STATE, FAIL CLOSED
Failure codes
| Code | Means | Typical cause |
TARGET_NOT_FOUND | No candidate matched | Screen isn't what we expect |
TARGET_AMBIGUOUS | Multiple candidates; refused | Ambiguous UI or loose descriptor |
PRECONDITION_FAILED | Step entry condition unmet | Prior step didn't land |
CHECKPOINT_FAILED | Flow complete, success false | Happy path not reached |
ACTION_FAILED | Driver couldn't perform action | Control disabled, obscured |
TIMEOUT | Wait never satisfied | Slow load or impossible condition |
NAVIGATION_BLOCKED | Allowlist violation | Off-domain redirect |
POLICY_DENIED | Policy refused the action | Irreversible without approval |
RECOVERY_EXHAUSTED | Outlived max_attempts | Persistent, not transient |
INPUT_VALIDATION_FAILED | Inputs failed declared schema | Caught before touching UI |
UNEXPECTED_STATE | Matched nothing declared | Fail-closed default |
LEASE_LOST | Stale lease epoch | Automation after human takeover |
The fault matrix
| Injected fault | Expected RunStatus | Expected code |
| member_id with no record | BUSINESS_OUTCOME | member_not_found |
| transient_load (502 once) | SUCCESS | recovered after retry |
| transient_load (502 always) | NEEDS_HUMAN | RECOVERY_EXHAUSTED (an escalation trigger of the reference capability) |
| session_timeout | SUCCESS | re-authenticated by run_capability corebank.auth.sign_on; escalates as RECOVERY_EXHAUSTED if that cannot run |
| undeclared interstitial | NEEDS_HUMAN | UNEXPECTED_STATE |
| validation_error on submit | BUSINESS_OUTCOME | validation_rejected |
| permission-denied account | BUSINESS_OUTCOME | permission_denied |
| off-allowlist redirect or frame navigation (refused before it is fetched) | FAILED | NAVIGATION_BLOCKED |
Control Transfer (Human-in-the-Loop)
A human takes control of the same live session the automation was using — not a fresh one — does the manual work, and hands it back so the run can finish.
State machine
RUNNING ──escalate──► PAUSED ──human_claim──► HUMAN_CONTROL
▲ (epoch++) (epoch++)
│ │
└── resume (epoch++) ◄── RESUMING ◄── human_release (epoch++)
Re-anchoring happens after the lease is back in RUNNING: ReplayExecutor.resume() reconciles against the live screen before its first action. cua demo drives that resume; the operator console's Release returns the lease to automation but does not itself restart the run.
The lease
Lease(holder: Actor, epoch: int, expires_at: datetime)
holder ∈ {AUTOMATION, HUMAN, SYSTEM} — SYSTEM while the session is PAUSED or RESUMING, when nobody may act. Every transition increments epoch monotonically.
An operator's input is refused by the broker unless lease.assert_held(HUMAN, epoch) passes. An automation run checks that it holds the session when it starts (or resumes), then carries its epoch into every dispatch, and the dispatcher refuses a stale one with LEASE_LOST. Every change of holder advances the epoch, so the two checks together cover every dispatch.
Escalation triggers
| Trigger | Meaning |
TARGET_AMBIGUOUS | Resolver refused to guess |
TARGET_NOT_FOUND | Ladder exhausted |
CHECKPOINT_FAILED | Reached the end, didn't reach the state |
RECOVERY_EXHAUSTED | A transient condition wasn't transient |
UNEXPECTED_STATE | Fail-closed default |
POLICY_DENIED | Policy refused an action (e.g. an irreversible step with no approval) — only if the capability lists it; the default triggers do not |
Re-anchoring after handoff
On human_release the broker snapshot-diffs pre-handoff against post-handoff and records human_delta. When the run is then resumed, ReplayExecutor.resume() scans forward from the step that escalated:
- A step whose postcondition already holds is skipped — doesn't redo human work. A step with no postcondition is never skipped.
- The run resumes at the first step not visibly done — provided its precondition holds
- Otherwise →
UNEXPECTED_STATE, fails closed
Step 3 is the fail-closed rule applied to the handoff seam. "The human left the session somewhere I don't recognize" is exactly the moment not to improvise.
ADR 0001 — UiSnapshot as the cross-surface abstraction
Status: Accepted · 2026-09-11
Context
We must perceive and act on three very different kinds of surface: modern web apps, legacy server-rendered web apps (framesets, nested tables, non-semantic markup, no test IDs), and native desktop applications.
Three candidates: DOM/CSS first (breaks on desktop), pure vision (pixel coordinates are the least stable thing you can persist), and semantic/accessibility tree.
Decision
Perception normalizes every surface into a UiSnapshot: a tree of UiNode {node_id, role, name, value, states, bounds, scope, parent, hints}.
The artifact schema and targeting system depend only on UiSnapshot and TargetDescriptor. Neither knows what an accessibility tree, a CDP session, or a DOM is.
| Driver | Populates UiSnapshot from |
PlaywrightCdpDriver (built) | CDP Accessibility.getFullAXTree + box model, enriched with DOM-derived hints |
DesktopUiaDriver (stubbed) | Windows UI Automation element tree |
| Future OCR driver | Vision model + OCR over a screenshot |
Consequences
Good. The same artifact can replay on a desktop driver, because role: button, name: "Search" means the same thing in UIA as in ARIA.
Bad. Accessibility trees on genuinely hostile legacy apps can be thin. Mitigated by driver-level enrichment and the structural_anchor resolution strategy.
ADR 0002 — Immutable content-addressed capability
Status: Accepted · 2026-09-11
Context
A capability artifact accumulates pressure to hold more: run counts, stability scores, approval state, last-run timestamps. An earlier draft put telemetry inside the capability. Two problems surfaced:
- The definition mutates every time you run it — git diffs become noise
- "Which version produced that run?" becomes unanswerable
Decision
A Capability at id@version is frozen and content-hashed. Nothing that changes as a result of executing it may live inside it. Runtime state moves into separate documents.
Consequences
Good. Reviewing a capability is reviewing a contract. A RunRecord can name exactly what it executed.
Also good. Immutability makes the multi-tenant story cleaner: a tenant cannot "drift" a shared capability by using it.
ADR 0003 — Four-class observation taxonomy
Status: Accepted · 2026-09-11
Context
A replay observes a screen and must decide what it means. The brief asks us to distinguish business outcomes, recoverable conditions, and hard failures. Working through the mock app exposed a fourth case: a state this capability does not understand.
A system without this category has to force it into one of the other three. The tempting choice is "keep going and see." That is precisely the behaviour that causes incidents.
Decision
Two distinct type families:
ObservationClass = EXPECTED | BUSINESS_OUTCOME | RECOVERABLE | HARD_FAILURE | UNEXPECTED_STATE
RunStatus = SUCCESS | BUSINESS_OUTCOME | NEEDS_HUMAN | FAILED
Consequences
Good. The caller's contract is honest: BUSINESS_OUTCOME(member_not_found) exits 0 and returns data. A FAILED carries enough detail to fix without reproducing.
Bad. Authoring a capability requires declaring outcomes and recovery rules explicitly — more work than letting the executor improvise. That is the point.
ADR 0004 — A single, type-enforced policy chokepoint
Status: Accepted · 2026-09-11
Context
Three different actors want to act on the live session. An earlier draft forwarded human clicks directly via CDP — which meant the moment a human took over, the allowlist, risk classification, and evidence trail all stopped applying.
Decision
One path to a surface, for every actor: Action → TargetResolver → PolicyEngine → SurfaceDriver.
Enforced three independent ways (import rule, type rule, reconciliation test), because an invariant everyone agrees with is weaker than one nobody can violate.
Consequences
Good. Discovery, replay, and human intervention are the same code path, differing only in who originates the action — so the guardrails we test are the guardrails that run.
Bad. raw_input is awkward to police semantically. A click is hit-tested against control geometry and a key against the focused control, so a click on “Transfer Funds” needs explicit confirmation; a gesture on something the semantic tree does not expose falls back to its action type’s tier.
ADR 0005 — Tenants overlay a capability; they never fork it
Status: Accepted · 2026-09-11
Context
The economics of this system: if automating a flow for one credit union means re-recording it for the next 200, the product does not work.
Decision
Capabilities are namespaced by vendor product, not by tenant. Per-tenant differences are expressed as overlay documents.
capability_ref: "corebank.member.savings_balance@1.2.0"
tenant: first-valley-cu
vars: { base_url: "https://fvcu.internal" }
overrides:
steps:
enter_member_id:
target: { name: { match: exact, value: "Member #" } }
Drift is measured, not assumed. Descending the resolution ladder at replay is a drift signal, not a silent success.
Consequences
Good. A rebranded label costs a four-line overlay, not a re-record. The drift score tells us which tenants are diverging before replays start failing.
Requirements Traceability
Every requirement mapped to the module that satisfies it, the test that proves it, and the evidence that demonstrates it.
A row with no test is a claim. A row with no evidence is a demo that never ran.
3.1 — Capability discovery
| Requirement | Module | Test |
| Accept a goal + target surface | cli/main.py, agent/loop.py | test_the_loop_completes_a_real_goal |
| Observe → decide → act against live surface | agent/loop.py | test_the_loop_completes_a_real_goal |
| Model sees semantic view, not raw HTML | agent/render.py | test_the_model_is_shown_a_snapshot_not_raw_html |
| Discovery obeys same guardrails as replay | runtime/dispatcher.py | test_discovery_actions_pass_through_the_chokepoint |
3.2 — The capability artifact
| Requirement | Module | Test |
| Ordered steps | domain/capability.py | test_reference_artifact_validates |
| Typed input parameters | domain/capability.py | test_the_literal_from_the_goal_becomes_a_parameter |
| Typed outputs | replay/executor.py | test_outputs_are_typed_from_what_was_observed |
| Versioned, immutable, reviewable | domain/capability.py | test_capability_is_frozen, test_tampering_is_detected |
| Callable by agent as typed function | domain/capability.py | test_exports_an_agent_callable_tool_schema |
3.3 — Deterministic replay
| Requirement | Module | Test |
| Replay without the LLM | replay/ | test_replay_uses_no_model + .importlinter |
| Ambiguity is refused, never guessed | targeting/resolver.py | test_ambiguity_is_refused_not_tiebroken |
| Business outcome ≠ failure | replay/classify.py | test_business_outcomes_are_answers_not_failures |
| Unknown states fail closed | replay/classify.py | test_an_undeclared_screen_fails_closed |
| Determinism | replay/executor.py | test_repeated_replays_produce_identical_decisions |
3.4 — Safety & data handling
| Requirement | Module | Test |
| Allowlist, enforced for every actor | policy/allowlist.py | test_allowlist_decisions |
| Nothing reaches a surface unauthorized | runtime/dispatcher.py | invariants/test_policy_chokepoint.py |
| Never persist secrets or raw PII | policy/redact.py | invariants/test_redaction.py |
3.6 — Human-in-the-loop escalation
| Requirement | Module | Test |
| A human takes the live session | hitl/broker.py | test_a_human_takes_over_acts_and_hands_back |
| Human input cannot bypass policy | hitl/broker.py | test_human_input_goes_through_policy |
| Stale automation cannot act after takeover | hitl/lease.py | test_stale_automation_cannot_act_after_a_takeover |
| Automation cannot act while human holds session | replay/executor.py | test_automation_cannot_act_after_a_human_claims_the_live_session |
Testing Strategy
Every architectural claim has a test that fails when the claim stops being true.
A design document that asserts an invariant is a wish. A contract in .importlinter plus a test that breaks on violation is an invariant.
Claim → test
| Claim | Test | Kind |
| Replay never calls the LLM | invariants/test_import_contracts.py, integration/test_fault_matrix.py::test_replay_uses_no_model + .importlinter | static + runtime |
| Policy chokepoint unbypassable | invariants/test_policy_chokepoint.py | static + runtime |
| Human control cannot bypass policy | invariants/test_human_control_safety.py | runtime |
| Stale automation refused | integration/test_handoff.py::test_automation_cannot_act_after_a_human_claims_the_live_session, integration/test_dispatcher.py::test_a_stale_lease_epoch_is_refused_before_anything_else | integration |
| Ambiguous targets refused | unit/test_resolver.py::test_ambiguity_is_refused_not_tiebroken | unit |
| Unknown states fail closed | integration/test_fault_matrix.py::test_an_undeclared_screen_fails_closed | integration |
| Replay is deterministic | integration/test_fault_matrix.py::test_repeated_replays_produce_identical_decisions | integration |
| Every sink is redacted | invariants/test_redaction.py | integration |
| Business outcomes ≠ failures | integration/test_fault_matrix.py | integration |
| Targeting is surface-neutral | integration/test_variant_b.py | integration |
| Capabilities immutable | contract/test_capability_schema.py::test_capability_is_frozen, ::test_tampering_is_detected | contract |
| Irreversible steps run only under a verified, scoped approval | integration/test_irreversible_approval.py | integration |
run_capability remedies go through the chokepoint and are bounded | integration/test_capability_remedy.py | integration |
domain/ is pure | .importlinter:pure-domain | static |
Test layers
| Layer | Proves | Cost |
unit/ | Pure logic: schema, resolver, scoring, policy, redaction, lease, taxonomy | free |
integration/ | Full discovery → artifact → replay against mock app, via fake LLM | free, deterministic |
contract/ | Schema stability: every artifact validates; JSON Schema export is golden | free |
invariants/ | The architectural claims above | free |
e2e/ | One real LLM run against a live browser | @pytest.mark.live, opt-in |
Offline by default
Everything except e2e/ runs with no API key and no network. 2,158 tests (1,800 of them edge cases: boundary, malformed, hostile and out-of-order input), mypy --strict clean. CI configures no API key on purpose — if a test needs the model, CI must fail rather than silently skip.
Continuous integration
.github/workflows/ci.yml runs the same targets a contributor runs locally, as separate steps so a failure names its stage:
- Lint and format, then strict type check
- Architectural invariants: the five import contracts, plus a check that every path the docs cite exists
- Secret scan (
make secrets HISTORY=1) over the tree and every commit. It never prints what it finds, since echoing a secret into CI logs is itself a leak
- The offline test suite, the reviewer-path smoke test, and the full 14-stage demo
The workflow holds read-only permissions, and a newer push cancels a superseded run on the same ref. CI has no model key on purpose. The demo's evidence is uploaded as a build artifact, so a reviewer can download exactly what that run produced.
A second workflow, .github/workflows/verify.yml, runs the slow checks CI does not run on every push: the edge-case runs (make edge-cases), the test report with line and branch coverage, and the mutation check, which breaks each invariant on purpose and fails if any deliberate violation goes uncaught. It runs nightly, on demand, and on pushes that touch code or tests, and uploads evidence/tests/ and evidence/edge-cases/ as an artifact. Dependabot keeps the pinned toolchain and the Actions versions current, weekly and grouped.
The one thing CI cannot do on every commit is spend real tokens. A separate, manually triggered Live discovery workflow runs make discover against your gateway, then make verify-live, and uploads the evidence as a build artifact to review before anything is committed. It needs an OMNIROUTE_API_KEY secret, plus CUA_LLM_BASE_URL and CUA_LLM_MODEL variables, and the gateway must be reachable from a GitHub-hosted runner.
The fake LLM
agent/fake.py implements LlmPort by replaying a recorded transcript. This exercises the real discovery loop — same policy, same resolver, same evidence bus — for free.
Risk: fixture drift. Mitigated by regenerating fixtures from a live run via script, plus a contract test asserting fixture shape matches the live LlmPort schema.
Run a Discovery
Discovery is the probabilistic half: an LLM drives a live UI to a goal and we record what it did. It is the expensive path, run once per capability.
Prerequisites
cp .env.example .env
make app
Run
cua discover \
--goal "Look up member 12345 and read their current savings balance" \
--target http://localhost:8811 \
--out evidence/capabilities/
What happens
- The driver observes the surface and normalizes it into a
UiSnapshot
- The snapshot is rendered for the model — not raw HTML, and redacted on the way out
- The model emits a tool call from the closed action space. It cannot invent an action
PolicyEngine authorizes it. Off-allowlist or irreversible actions are blocked
- The resolver resolves the target; the dispatcher dispatches; the bus records everything
- Loop until the goal is met or a stop condition fires
- On success, the compiler emits a draft Capability
Output
evidence/discovery/<run_id>/
trace.jsonl authorize / resolve / dispatch / observe, actor-tagged;
each llm_call carries the gateway's provider and request id
and the measured latency_ms;
run_end carries the stop reason and budget (steps, tokens, elapsed)
snapshots/ UiSnapshot per step
screenshots/ redacted
run_record.json timings, model, tokens_used -- summed from the llm_call events,
and equal to the budget's token count
evidence/capabilities/<id>@<version>.yaml
Verify it was live
make verify-live
make verify-live RUN=evidence/discovery/<run_id>
Hard checks fail the command: a named, non-scripted model, tokens spent, and a gateway-issued request id on every model call. A median latency under 100 ms only warns, because a hosted model rarely answers that fast but a local gateway with a warm cache can. Runs recorded before latency_ms existed fall back to timestamp gaps and are labelled derived, not measured.
If it fails
| Symptom | Look at |
verify-live says the run was scripted | No OMNIROUTE_API_KEY was set, so discovery fell back to the recorded transcript (model_name: fake/scripted) |
Stops at max_steps | the llm_call events in trace.jsonl — is it looping on one screen? |
POLICY_DENIED | Expected if the goal needs an irreversible action |
TARGET_AMBIGUOUS | The screen genuinely has two matching controls. Good — it refused. |
| Model can't find a control | Check the UiSnapshot in snapshots/ |
Run a Replay
Replay is the deterministic half: the production execution path. No model, no decisions, no surprises.
make app
cua replay evidence/capabilities/corebank.member.savings_balance@1.0.0.yaml \
--input member_id=67890
Exercising error paths
Faults are armed on the mock app out of band, never through cua replay (it has no fault flag): POST /_control/arm with {"fault": <name>, "count": <n>} (-1 = every request), after signing in. cua console --capability <ref> --arm-fault <name> does the same for a supervised run.
| Input / armed fault | Result |
--input member_id=99999 | BUSINESS_OUTCOME(member_not_found) |
transient_load, count 1 | RECOVERABLE → recovered → SUCCESS |
transient_load, count -1 | RECOVERY_EXHAUSTED → escalates (NEEDS_HUMAN) |
undeclared_dialog | UNEXPECTED_STATE → fails closed |
session_timeout | re-authenticated by run_capability corebank.auth.sign_on → SUCCESS; escalates if that cannot run |
The session_timeout remedy needs corebank.auth.sign_on in the catalog and the operator credentials as secrets (CUA_SECRET_COREBANK_OPERATOR_ID, CUA_SECRET_COREBANK_OPERATOR_PASSWORD). Without either it fails closed as RECOVERY_EXHAUSTED.
Debugging a failure
| Failure | Usual cause |
TARGET_NOT_FOUND | The screen isn't what the artifact expects |
TARGET_AMBIGUOUS | Descriptor too loose, or genuinely two matches |
CHECKPOINT_FAILED | Steps ran but goal state wasn't reached |
UNEXPECTED_STATE | Capability doesn't declare this screen |
RECOVERY_EXHAUSTED | The "transient" condition wasn't transient |
Determinism
Same artifact + same inputs + same app state ⇒ identical decision trace. If two replays diverge, that is a bug in this system, not flakiness.
Handle an Escalation
A run stopped because it could not safely proceed. This is a designed outcome, not a crash.
Taking control
cua console
Claim takes the lease. The lease epoch increments, and any in-flight automation dispatch is refused with LEASE_LOST.
You are driving the same live session, not a copy.
What you can and cannot do
| You, during an intervention |
| Domain allowlist | Still enforced |
| Ordinary actions | Allowed, recorded |
| Irreversible actions | Allowed with explicit confirmation, recorded |
| Evidence | Everything captured, tagged actor=HUMAN |
Handing back
Release diffs session state, records human_delta, and returns the lease to automation. The console does not restart the run itself; when a caller resumes it (ReplayExecutor.resume(), as cua demo does), the executor:
- Scans forward from the step that escalated
- Skips a step only if its postcondition already holds (a step with no postcondition is never skipped)
- Resumes at the first step not visibly done, provided its precondition holds
- Otherwise stops with
UNEXPECTED_STATE
Afterwards
A recurring escalation is a capability defect, not an operations problem. Usual fixes:
- Screen seen repeatedly but undeclared → add an
outcomes or recovery entry
TARGET_AMBIGUOUS on the same step → tighten the descriptor
- Rising
drift_score for one tenant → add a TenantBinding overlay, or re-record