Documentation

Documentation for the Computer-Use Automation system. Every statement describes what is built, or is marked explicitly as planned. New here? Start with Quick Start, then Demo Walkthrough.

Getting Started

What this is, how to run it, and how to configure a model — including a local one.

  • Quick Startsetup + dependencies
  • Demo Walkthrough14 stages, end to end
  • CLI Referenceevery command

Architecture

How the system is put together, and why the boundaries fall where they do.

  • System Architecturepackage map + dependency rules
  • Execution PipelineAction → Resolver → Policy → Driver
  • Ten Invariantsmechanically enforced contracts

Design

Detailed design documents — the artifact schema, target resolution, error taxonomy, and human-in-the-loop control transfer.

  • Capability Artifactdesign/artifact-schema
  • Target Resolutiondesign/target-resolution
  • Error Taxonomydesign/error-taxonomy
  • Control Transfer (HITL)design/control-transfer

Decisions (ADRs)

Why it is this way — the architectural decision records.

  • ADR 0001 — UiSnapshot as cross-surface abstractionaccepted
  • ADR 0002 — Immutable content-addressed capabilityaccepted
  • ADR 0003 — Four-class observation taxonomyaccepted
  • ADR 0004 — Single policy chokepointaccepted
  • ADR 0005 — Tenant overlay, not forkaccepted

Product

  • Requirements Traceabilityrequirements-traceability

Testing

  • Testing Strategyclaim → test table

Runbooks

Step-by-step operating procedures.

  • Run a Discoveryrunbooks/run-a-discovery
  • Run a Replayrunbooks/run-a-replay
  • Handle an Escalationrunbooks/handle-an-escalation

Quick Start

Get the system running in under a minute.

Prerequisites

  • Python 3.11+
  • uv (optional, used if present; otherwise venv + pip)

Setup

# Clone and enter the project
git clone https://github.com/adityamhaske/computer-use-automation.git
cd computer-use-automation

# Install everything: venv, dependencies, Chromium
make setup

Configuration

cp .env.example .env

The only thing that needs an API key is the live discovery run. Replay, escalation, the full test suite, and the demo all work without one.

VariablePurpose
OMNIROUTE_API_KEYDiscovery only. Any OpenAI-compatible gateway works; OPENROUTER_API_KEY is still read as a fallback. Pair with CUA_LLM_BASE_URL and CUA_LLM_MODEL.
CUA_LLM_BASE_URLThe gateway's /v1 endpoint (OmniRoute). Blank falls back to OpenRouter.
CUA_LLM_MODELModel ID for discovery. Name a concrete model: a router alias such as auto hides which one ran.
CUA_POLICY_FILEAllowlist and risk policy — defaults to config/policy.yaml

Run the demo

# The whole story, one command — fourteen stages end to end
make demo

Takes about a minute. Works without an API key — discovery falls back to a recorded transcript and says so explicitly.

Unified Orchestrator (./start.sh)

Orchestration over the Makefile targets — it adds nothing you could not run by hand, it just names the combinations (app and console together, wait-for-ready, teardown on Ctrl+C).

./start.sh                 # Default: setup venv + run the full 14-stage demo
./start.sh ui              # Console + mock bank, with an escalation already waiting
./start.sh ui --headed     # ...and watch the supervised browser while you claim it
./start.sh ui --bare       # Console over an idle session: the queue stays empty
./start.sh app             # Run hostile mock back-office app (port 8811)
./start.sh app-b           # Run rebranded second-tenant variant (port 8811)
./start.sh check           # Run CI checks: lint + typecheck + invariants + secret scan + tests + smoke
./start.sh test            # Run offline test suite (no API keys, no network)
./start.sh eval            # Run stability and cross-tenant drift evaluations

Development commands (Makefile)

make check          # lint + typecheck + invariants + secret scan + tests + smoke (CI)
make test           # offline suite; no API key, no network
make invariants     # just the architectural contracts
make secrets        # fail on any key-shaped string in tracked files (HISTORY=1 adds every commit)
make discover       # live LLM discovery against the mock app; needs OMNIROUTE_API_KEY
make verify-live    # check the newest discovery run was a live model, not a script
make eval           # stability + cross-tenant measurement
make edge-cases     # 22 hostile/boundary scenarios run for real, every run saved to evidence/edge-cases/
make test-report    # whole suite under line + branch coverage -> evidence/tests/
make mutation-check # break each invariant on purpose; prove the suite goes red
make evidence       # refresh everything under evidence/ in one go

2,158 tests, mypy --strict clean, five enforced import contracts. The whole suite runs offline with no API key. make test-report writes the counts and coverage to evidence/tests/; make mutation-check breaks each invariant on purpose and proves the suite goes red.

Demo Walkthrough

make demo boots the mock back-office and runs fourteen stages end to end, writing everything to evidence/.

#StageWhat happensResult
1Mock back-office runningHostile frameset app, no test IDsREADY
2DiscoveryThe model drives the live UI to the goalTRACE
3Capability compiled and sealedTyped I/O, semantic targets, content hashSEALED
4Deterministic replay, new inputsNo model in the loopSUCCESS
5Business outcomemember_not_found — an answer, not a crashEXIT 0
6Injected 502Declared recovery cleared itSUCCESS
7Fault past budgetRecovery exhausted → escalateNEEDS_HUMAN
8Malformed inputRejected before actingFAILED
9Undeclared screenFailed closed and escalatedNEEDS_HUMAN
10Operator took overDrove the same live session, handed it backEPOCH 3
11Re-anchored after handoffResumes at the right step, not step 1SUCCESS
12Agent invocationCalled by name with typed args and declared outcomesSUCCESS
No API key required. Stage 2 falls back to a recorded transcript and says so, in the output and in the evidence. Everything after it is model-free by construction. Set OMNIROUTE_API_KEY for a genuine LLM-driven discovery, then make verify-live to check the run was live.

Fault injection

Faults are injected into the mock app, not the replay engine — the executor must not know a fault is coming:

# Arm a fault
curl -X POST localhost:8811/_control/arm \
  -d '{"fault":"transient_load","count":1}' \
  -H 'content-type: application/json'

# Reset
curl -X POST localhost:8811/_control/reset

Available faults: transient_load, session_timeout, undeclared_dialog, validation_error.

Exit codes

StatusExitMeaning
SUCCESS0Checkpoint verified, typed outputs returned
BUSINESS_OUTCOME0A legitimate answer — "no such member" is not a failure
NEEDS_HUMAN2Stopped safely; an intervention is queued
FAILED1A defect. Carries step, expected, observed.

CLI Reference

make demo is orchestration, not a second implementation — every stage is a command you can run on its own.

Discovery

cua discover \
  --goal "Look up member 12345 and read their current savings balance" \
  --target http://localhost:8811 \
  --out evidence/capabilities/

Requires OMNIROUTE_API_KEY. The model drives a live UI, the run is compiled into a capability artifact.

Replay

# Success case
cua replay <artifact> --base-url http://localhost:8811 --input member_id=67890

# Business outcome (exit 0)
cua replay <artifact> --base-url http://localhost:8811 --input member_id=99999

Operator Console

cua console --target http://localhost:8811    # claim, act, release

Catalog

cua catalog list                              # typed signatures
cua catalog show <id> --tool-schema           # agent-facing contract
cua catalog invoke <id> --input member_id=12345 --base-url http://localhost:8811

Agent Demo

cua agent-demo --base-url http://localhost:8811   # caller handling all four statuses

Mock App

make app                  # hostile back-office on :8811
make app-variant-b        # second tenant variant

System Architecture

How the pieces fit together, and why the boundaries fall where they do.

Package map

PackageOwnsMay import
cua/domainTypes: Capability, TargetDescriptor, Action, UiSnapshot, RunRecord, taxonomynothing from cua
cua/perceptionNormalizing raw surface observation into UiSnapshot; fingerprintingdomain
cua/targetingResolution ladder, scoring, ambiguity, driftdomain, perception
cua/policyAllowlist, risk tiers, redaction, secrets, AuthorizedActiondomain, targeting
cua/hitlLease, SessionBroker, intervention queue, operator consoledomain, policy
cua/runtimedispatcher.py — the sole importer of surfaceseverything below
cua/surfacesSurfaceDriver port; Playwright/CDP driver; desktop stubdomain, perception
cua/agentLlmPort, prompts, the discovery loopruntime and below
cua/replayThe deterministic executorruntime and below — never agent
cua/recorderCompiling a discovery trace into a Capabilitydomain, targeting
cua/evidenceStructured, redacted, actor-tagged run recordsdomain, policy
cua/catalogCapability store, tenant overlays, tool-spec exportdomain
cua/cliEntry pointseverything

Project layout

src/cua/          domain · perception · surfaces · targeting · policy · runtime
                  agent · recorder · replay · hitl · evidence · cli
apps/mock_bank/   the hostile target application (+ a "second tenant" variant)
docs/             ADRs (why) · design (how) · prd (what, in what order) · runbooks
tests/            unit · integration · contract · invariants · e2e(live)
evidence/         proof that the end-to-end thread actually ran
                  including evals/ — the measured numbers behind the claims

Execution Pipeline

One path to a surface, for every actor. Discovery, replay, and human intervention all take the same path.

Action ──► TargetResolver ──► PolicyEngine ──► SurfaceDriver
       (resolve or refuse)    (authorize)      (dispatch)

Enforced three independent ways:

  1. Import rule. cua.runtime.dispatcher is the only module allowed to import cua.surfaces (.importlinter contract policy-chokepoint).
  2. Type rule. SurfaceDriver.dispatch() accepts only an AuthorizedAction, which carries a token instance held exclusively by PolicyEngine.
  3. Reconciliation test. Every dispatch event in a run record must have a matching authorize event with the same action ID.

Human input is policed, not injected

The console submits raw_input actions which traverse the same path under a distinct HUMAN policy profile:

AUTOMATIONHUMAN (during intervention)
Domain allowlistenforcedenforced
safe actionsallowallow
elevated actionsallow if declared elevated by the stepallow, recorded
irreversible actionsrefused, unless the step is declared irreversible and the run carries an approval of that exact content plus the caller's opt-in (allow_if_approved, checked at dispatch)allow with confirmation, recorded
Off-artifact navigationNAVIGATION_BLOCKEDallowed within allowlist, recorded
Evidence redactionappliedapplied

The Ten Invariants

The value of this system is almost entirely in a handful of properties. Each is mechanically enforced — the test is named so you can see it fail on purpose.

1. Replay never calls or imports the LLM

cua.replay may not import cua.agent, httpx, or any model client — directly or transitively. A replay that can quietly ask a model for help is not deterministic, not cheap, and not auditable.

2. No action reaches a driver without policy authorization

cua.runtime.dispatcher is the only module permitted to call dispatch(). SurfaceDriver.dispatch() accepts only an AuthorizedAction, which carries a token that only PolicyEngine can mint.

3. Target identity is a semantic TargetDescriptor, never a raw CSS selector

Artifacts describe controls by role, accessible name, scope, structural anchor, and ordinal. Surface-specific data may appear only inside TargetDescriptor.hints, and a hint is an unverified cache.

4. Human input cannot bypass policy or evidence

The operator console never injects into the page. It submits raw_input actions to the SessionBroker, which runs them through the same chokepoint under the HUMAN policy profile.

5. Unknown states fail closed

If the observed UI matches no declared precondition, checkpoint, outcome, or recovery rule, it is UNEXPECTED_STATE and the run stops. If a target resolves ambiguously, the resolver refuses with TARGET_AMBIGUOUS.

6. Every sink is redacted

Logs, artifacts, evidence files, screenshots, and outbound LLM prompts. Secrets are {$secret: ref} references resolved at dispatch time and never written anywhere.

7. Business outcomes are not failures

"No such member" is a legitimate answer the caller needs, not a crash. BUSINESS_OUTCOME exits 0.

8. Automation may only act while it holds the session

Every dispatch asks the lease: who holds this session, and has it changed hands since this run was authorized. Either answer can refuse the dispatch with LEASE_LOST.

9. Capabilities are immutable

A Capability at id@version is frozen and content-hashed. Anything that changes because you ran it lives in a separate document: RunRecord, CapabilityEvaluation, CapabilityApproval.

10. domain/ is pure

No I/O, no network, no browser, no clock. cua.domain imports nothing else from cua.

The Capability Artifact

The unit of reuse in this system. What the LLM's discovery run compiles down to, what a human reviews, and what an AI agent invokes in production.

Decision rationale lives in ADR 0002. This document is the reference for what the schema is.

Three readers, one artifact

ReaderNeeds
Replay engineUnambiguous, executable steps with explicit success and failure conditions
Human reviewerTo understand what it does, what it touches, and what could go wrong — without running it
Calling AI agentA typed contract: what to pass, what comes back, what can happen

Document family

Only the first is the capability. The rest exist so the capability can stay immutable.

cua.capability/v1             frozen, content-hashed — the contract
cua.tenant_binding/v1         per-tenant vars + step overrides (base ⊕ overlay)
cua.run_record/v1             one execution: trace, resolutions, drift, evidence refs
cua.capability_evaluation/v1  aggregate over runs: stability, strategy mix, drift trend
cua.capability_approval/v1    draft → approved, approver, scope

Identity

capability:
  id: "corebank.member.savings_balance"   # <vendor-product>.<domain>.<operation>
  version: "1.2.0"
  content_hash: "sha256:…"

id is namespaced by vendor product, never tenant — the precondition for cross-tenant reuse.

Typed contract

inputs:
  - { name: member_id, type: string, required: true, pattern: "^[0-9]{4,10}$",
      sensitive: false, example: "12345" }
outputs:
  - { name: savings_balance, type: money, source: { step: read_balance } }

These emit JSON Schema directly, so the artifact is the agent tool contract — there is no second source of truth to drift. sensitive: true propagates everywhere.

Steps

steps:
  - id: enter_member_id
    action: { type: type, value: { $input: member_id } }
    target:
      role: textbox
      name: { match: exact, value: "Member Number" }
      scope: { frame: content, region: search_form }
      anchor: { relation: row_of, cell_text: "Member Number" }
      hints: { css: "...", node_path: "..." }   # unverified cache
      fingerprint: "sha256:…"
      recorded_strategy: semantic_exact
    precondition:  { assert: node_exists, role: textbox, name: "Member Number" }
    wait:          { for: snapshot_stable, timeout_ms: 5000 }
    postcondition: { assert: node_has_value, value: { $input: member_id } }
    risk: safe

Every step asserts both before and after. A step that only acts is a step that assumes the click worked.

Success, outcomes, recovery

# Did we actually reach the state we wanted?
checkpoint:
  all_of:
    - { assert: node_exists, role: heading, name_contains: "Member Detail" }
    - { assert: node_matches, role: cell, name_matches: '^\\$[0-9,]+\\.[0-9]{2}$' }

# Legitimate answers — NOT failures
outcomes:
  - { code: member_not_found, detect: { text_present: "No records found" },
      terminal: true, returns: { member_id: { $input: member_id } } }

# Bounded, declared remediation — never open-ended
recovery:
  - { id: transient_load, detect: { http_status_in: [502,503,504] },
      remedy: [ { reload: {} } ], max_attempts: 3, backoff: exponential }

Target Resolution

Resolution is where determinism is won or lost. Turning a semantic TargetDescriptor into a concrete UiNode in the current snapshot.

The resolution ladder

Strategies are tried in fixed order. The first rung that yields any candidate decides: exactly one wins; more than one is refused unless the descriptor carries an explicit ordinal. A rung with several candidates does not fall through to the next.

#StrategyMatches onSurvives
1semantic_exactrole + accessible name, exact, unique in scopemost things
2semantic_normalizedsame, case/whitespace/punctuation foldedcosmetic label edits
3structural_anchorposition relative to nearby textcontrols with no usable name at all
4hint_cachedstored CSS/node-path hintnothing structural; speed optimization
5ordinal_in_regionNth control of a role within a scoperebranding; last resort before refusal
6visionOCR / bounding boxnot implemented — no generator exists; replay excludes it by construction

Rung 3 is what makes legacy enterprise apps tractable. Those apps are full of controls with no name, no label, and no test ID, sitting in a table cell next to the text that identifies them.

Rung 4 is a cache, not an identity. A hint is accepted only if the node it resolves to also satisfies the semantic assertion.

Ambiguity is refusal

If a rung yields two or more candidates and the descriptor carries no explicit ordinal, the resolver raises TARGET_AMBIGUOUS and the run stops. It does not pick the first, the topmost, or the highest-scoring. There is no threshold to tune: the reported ambiguity figure, (candidates − 1) / candidates, is for debugging and is never compared against anything.

Two plausible "Submit" buttons mean the screen is not what we think it is, and acting on either is how an automation posts a transaction to the wrong account. A refusal is recoverable — a wrong click may not be.

Drift

Resolving via a lower-priority strategy than the one recorded at discovery is drift — a signal, not a silent success. Per-step drift aggregates into a run-level drift_score.

This is the mechanism behind the multi-tenant story: a tenant whose UI has diverged shows rising drift before replays begin to fail.

Variant B: two kinds of divergence

Case A — markup churn, same vocabulary (the ladder handles this)

BaseVariant B
accessible nameMember NumberMember Number — same
form field namememnomember_num — changed
CSS classfrmfldng-inp — changed

semantic_exact resolves it on both. This is the falsifiable form of "this system does not depend on CSS selectors."

Case B — rebranding, different vocabulary (this needs an overlay)

BaseVariant B
balance labelSavings BalanceSavings Bal.
account typeAccount TypeType of Account

No automatic recovery here — and that is correct. Inferring that "Savings Bal." means "Savings Balance" is a fuzzy guess, and a system that guesses which row holds a balance will eventually read the wrong one.

Error Taxonomy

Two distinct type families — what we observed, and how the run ended — because they are not the same axis.

What we observed (a classification of a moment)

ObservationClass = EXPECTED | BUSINESS_OUTCOME | RECOVERABLE | HARD_FAILURE | UNEXPECTED_STATE

How the run ended (a terminal status)

RunStatus = SUCCESS | BUSINESS_OUTCOME | NEEDS_HUMAN | FAILED

The decision procedure

At every observation point the executor asks, in this order:

1. Does the state satisfy the step precondition / checkpoint?   → EXPECTED, proceed
2. Does it match a declared `outcomes` detector?                → BUSINESS_OUTCOME, return
3. Does it match a declared `recovery` detector?                → RECOVERABLE, remediate
4. Is it a known hard-failure condition?                        → HARD_FAILURE, stop
5. Otherwise                                                    → UNEXPECTED_STATE, FAIL CLOSED

Failure codes

CodeMeansTypical cause
TARGET_NOT_FOUNDNo candidate matchedScreen isn't what we expect
TARGET_AMBIGUOUSMultiple candidates; refusedAmbiguous UI or loose descriptor
PRECONDITION_FAILEDStep entry condition unmetPrior step didn't land
CHECKPOINT_FAILEDFlow complete, success falseHappy path not reached
ACTION_FAILEDDriver couldn't perform actionControl disabled, obscured
TIMEOUTWait never satisfiedSlow load or impossible condition
NAVIGATION_BLOCKEDAllowlist violationOff-domain redirect
POLICY_DENIEDPolicy refused the actionIrreversible without approval
RECOVERY_EXHAUSTEDOutlived max_attemptsPersistent, not transient
INPUT_VALIDATION_FAILEDInputs failed declared schemaCaught before touching UI
UNEXPECTED_STATEMatched nothing declaredFail-closed default
LEASE_LOSTStale lease epochAutomation after human takeover

The fault matrix

Injected faultExpected RunStatusExpected code
member_id with no recordBUSINESS_OUTCOMEmember_not_found
transient_load (502 once)SUCCESSrecovered after retry
transient_load (502 always)NEEDS_HUMANRECOVERY_EXHAUSTED (an escalation trigger of the reference capability)
session_timeoutSUCCESSre-authenticated by run_capability corebank.auth.sign_on; escalates as RECOVERY_EXHAUSTED if that cannot run
undeclared interstitialNEEDS_HUMANUNEXPECTED_STATE
validation_error on submitBUSINESS_OUTCOMEvalidation_rejected
permission-denied accountBUSINESS_OUTCOMEpermission_denied
off-allowlist redirect or frame navigation (refused before it is fetched)FAILEDNAVIGATION_BLOCKED

Control Transfer (Human-in-the-Loop)

A human takes control of the same live session the automation was using — not a fresh one — does the manual work, and hands it back so the run can finish.

State machine

RUNNING ──escalate──► PAUSED ──human_claim──► HUMAN_CONTROL
   ▲                  (epoch++)               (epoch++)
   │                                                │
   └── resume (epoch++) ◄── RESUMING ◄── human_release (epoch++)

Re-anchoring happens after the lease is back in RUNNING: ReplayExecutor.resume() reconciles against the live screen before its first action. cua demo drives that resume; the operator console's Release returns the lease to automation but does not itself restart the run.

The lease

Lease(holder: Actor, epoch: int, expires_at: datetime)

holder ∈ {AUTOMATION, HUMAN, SYSTEM} — SYSTEM while the session is PAUSED or RESUMING, when nobody may act. Every transition increments epoch monotonically.

An operator's input is refused by the broker unless lease.assert_held(HUMAN, epoch) passes. An automation run checks that it holds the session when it starts (or resumes), then carries its epoch into every dispatch, and the dispatcher refuses a stale one with LEASE_LOST. Every change of holder advances the epoch, so the two checks together cover every dispatch.

Escalation triggers

TriggerMeaning
TARGET_AMBIGUOUSResolver refused to guess
TARGET_NOT_FOUNDLadder exhausted
CHECKPOINT_FAILEDReached the end, didn't reach the state
RECOVERY_EXHAUSTEDA transient condition wasn't transient
UNEXPECTED_STATEFail-closed default
POLICY_DENIEDPolicy refused an action (e.g. an irreversible step with no approval) — only if the capability lists it; the default triggers do not

Re-anchoring after handoff

On human_release the broker snapshot-diffs pre-handoff against post-handoff and records human_delta. When the run is then resumed, ReplayExecutor.resume() scans forward from the step that escalated:

  1. A step whose postcondition already holds is skipped — doesn't redo human work. A step with no postcondition is never skipped.
  2. The run resumes at the first step not visibly done — provided its precondition holds
  3. Otherwise → UNEXPECTED_STATE, fails closed

Step 3 is the fail-closed rule applied to the handoff seam. "The human left the session somewhere I don't recognize" is exactly the moment not to improvise.

ADR 0001 — UiSnapshot as the cross-surface abstraction

Status: Accepted · 2026-09-11

Context

We must perceive and act on three very different kinds of surface: modern web apps, legacy server-rendered web apps (framesets, nested tables, non-semantic markup, no test IDs), and native desktop applications.

Three candidates: DOM/CSS first (breaks on desktop), pure vision (pixel coordinates are the least stable thing you can persist), and semantic/accessibility tree.

Decision

Perception normalizes every surface into a UiSnapshot: a tree of UiNode {node_id, role, name, value, states, bounds, scope, parent, hints}.

The artifact schema and targeting system depend only on UiSnapshot and TargetDescriptor. Neither knows what an accessibility tree, a CDP session, or a DOM is.

DriverPopulates UiSnapshot from
PlaywrightCdpDriver (built)CDP Accessibility.getFullAXTree + box model, enriched with DOM-derived hints
DesktopUiaDriver (stubbed)Windows UI Automation element tree
Future OCR driverVision model + OCR over a screenshot

Consequences

Good. The same artifact can replay on a desktop driver, because role: button, name: "Search" means the same thing in UIA as in ARIA.

Bad. Accessibility trees on genuinely hostile legacy apps can be thin. Mitigated by driver-level enrichment and the structural_anchor resolution strategy.

ADR 0002 — Immutable content-addressed capability

Status: Accepted · 2026-09-11

Context

A capability artifact accumulates pressure to hold more: run counts, stability scores, approval state, last-run timestamps. An earlier draft put telemetry inside the capability. Two problems surfaced:

  1. The definition mutates every time you run it — git diffs become noise
  2. "Which version produced that run?" becomes unanswerable

Decision

A Capability at id@version is frozen and content-hashed. Nothing that changes as a result of executing it may live inside it. Runtime state moves into separate documents.

Consequences

Good. Reviewing a capability is reviewing a contract. A RunRecord can name exactly what it executed.

Also good. Immutability makes the multi-tenant story cleaner: a tenant cannot "drift" a shared capability by using it.

ADR 0003 — Four-class observation taxonomy

Status: Accepted · 2026-09-11

Context

A replay observes a screen and must decide what it means. The brief asks us to distinguish business outcomes, recoverable conditions, and hard failures. Working through the mock app exposed a fourth case: a state this capability does not understand.

A system without this category has to force it into one of the other three. The tempting choice is "keep going and see." That is precisely the behaviour that causes incidents.

Decision

Two distinct type families:

ObservationClass = EXPECTED | BUSINESS_OUTCOME | RECOVERABLE | HARD_FAILURE | UNEXPECTED_STATE
RunStatus = SUCCESS | BUSINESS_OUTCOME | NEEDS_HUMAN | FAILED

Consequences

Good. The caller's contract is honest: BUSINESS_OUTCOME(member_not_found) exits 0 and returns data. A FAILED carries enough detail to fix without reproducing.

Bad. Authoring a capability requires declaring outcomes and recovery rules explicitly — more work than letting the executor improvise. That is the point.

ADR 0004 — A single, type-enforced policy chokepoint

Status: Accepted · 2026-09-11

Context

Three different actors want to act on the live session. An earlier draft forwarded human clicks directly via CDP — which meant the moment a human took over, the allowlist, risk classification, and evidence trail all stopped applying.

Decision

One path to a surface, for every actor: Action → TargetResolver → PolicyEngine → SurfaceDriver.

Enforced three independent ways (import rule, type rule, reconciliation test), because an invariant everyone agrees with is weaker than one nobody can violate.

Consequences

Good. Discovery, replay, and human intervention are the same code path, differing only in who originates the action — so the guardrails we test are the guardrails that run.

Bad. raw_input is awkward to police semantically. A click is hit-tested against control geometry and a key against the focused control, so a click on “Transfer Funds” needs explicit confirmation; a gesture on something the semantic tree does not expose falls back to its action type’s tier.

ADR 0005 — Tenants overlay a capability; they never fork it

Status: Accepted · 2026-09-11

Context

The economics of this system: if automating a flow for one credit union means re-recording it for the next 200, the product does not work.

Decision

Capabilities are namespaced by vendor product, not by tenant. Per-tenant differences are expressed as overlay documents.

# cua.tenant_binding/v1
capability_ref: "corebank.member.savings_balance@1.2.0"
tenant: first-valley-cu
vars: { base_url: "https://fvcu.internal" }
overrides:
  steps:
    enter_member_id:
      target: { name: { match: exact, value: "Member #" } }

Drift is measured, not assumed. Descending the resolution ladder at replay is a drift signal, not a silent success.

Consequences

Good. A rebranded label costs a four-line overlay, not a re-record. The drift score tells us which tenants are diverging before replays start failing.

Requirements Traceability

Every requirement mapped to the module that satisfies it, the test that proves it, and the evidence that demonstrates it.

A row with no test is a claim. A row with no evidence is a demo that never ran.

3.1 — Capability discovery

RequirementModuleTest
Accept a goal + target surfacecli/main.py, agent/loop.pytest_the_loop_completes_a_real_goal
Observe → decide → act against live surfaceagent/loop.pytest_the_loop_completes_a_real_goal
Model sees semantic view, not raw HTMLagent/render.pytest_the_model_is_shown_a_snapshot_not_raw_html
Discovery obeys same guardrails as replayruntime/dispatcher.pytest_discovery_actions_pass_through_the_chokepoint

3.2 — The capability artifact

RequirementModuleTest
Ordered stepsdomain/capability.pytest_reference_artifact_validates
Typed input parametersdomain/capability.pytest_the_literal_from_the_goal_becomes_a_parameter
Typed outputsreplay/executor.pytest_outputs_are_typed_from_what_was_observed
Versioned, immutable, reviewabledomain/capability.pytest_capability_is_frozen, test_tampering_is_detected
Callable by agent as typed functiondomain/capability.pytest_exports_an_agent_callable_tool_schema

3.3 — Deterministic replay

RequirementModuleTest
Replay without the LLMreplay/test_replay_uses_no_model + .importlinter
Ambiguity is refused, never guessedtargeting/resolver.pytest_ambiguity_is_refused_not_tiebroken
Business outcome ≠ failurereplay/classify.pytest_business_outcomes_are_answers_not_failures
Unknown states fail closedreplay/classify.pytest_an_undeclared_screen_fails_closed
Determinismreplay/executor.pytest_repeated_replays_produce_identical_decisions

3.4 — Safety & data handling

RequirementModuleTest
Allowlist, enforced for every actorpolicy/allowlist.pytest_allowlist_decisions
Nothing reaches a surface unauthorizedruntime/dispatcher.pyinvariants/test_policy_chokepoint.py
Never persist secrets or raw PIIpolicy/redact.pyinvariants/test_redaction.py

3.6 — Human-in-the-loop escalation

RequirementModuleTest
A human takes the live sessionhitl/broker.pytest_a_human_takes_over_acts_and_hands_back
Human input cannot bypass policyhitl/broker.pytest_human_input_goes_through_policy
Stale automation cannot act after takeoverhitl/lease.pytest_stale_automation_cannot_act_after_a_takeover
Automation cannot act while human holds sessionreplay/executor.pytest_automation_cannot_act_after_a_human_claims_the_live_session

Testing Strategy

Every architectural claim has a test that fails when the claim stops being true.

A design document that asserts an invariant is a wish. A contract in .importlinter plus a test that breaks on violation is an invariant.

Claim → test

ClaimTestKind
Replay never calls the LLMinvariants/test_import_contracts.py, integration/test_fault_matrix.py::test_replay_uses_no_model + .importlinterstatic + runtime
Policy chokepoint unbypassableinvariants/test_policy_chokepoint.pystatic + runtime
Human control cannot bypass policyinvariants/test_human_control_safety.pyruntime
Stale automation refusedintegration/test_handoff.py::test_automation_cannot_act_after_a_human_claims_the_live_session, integration/test_dispatcher.py::test_a_stale_lease_epoch_is_refused_before_anything_elseintegration
Ambiguous targets refusedunit/test_resolver.py::test_ambiguity_is_refused_not_tiebrokenunit
Unknown states fail closedintegration/test_fault_matrix.py::test_an_undeclared_screen_fails_closedintegration
Replay is deterministicintegration/test_fault_matrix.py::test_repeated_replays_produce_identical_decisionsintegration
Every sink is redactedinvariants/test_redaction.pyintegration
Business outcomes ≠ failuresintegration/test_fault_matrix.pyintegration
Targeting is surface-neutralintegration/test_variant_b.pyintegration
Capabilities immutablecontract/test_capability_schema.py::test_capability_is_frozen, ::test_tampering_is_detectedcontract
Irreversible steps run only under a verified, scoped approvalintegration/test_irreversible_approval.pyintegration
run_capability remedies go through the chokepoint and are boundedintegration/test_capability_remedy.pyintegration
domain/ is pure.importlinter:pure-domainstatic

Test layers

LayerProvesCost
unit/Pure logic: schema, resolver, scoring, policy, redaction, lease, taxonomyfree
integration/Full discovery → artifact → replay against mock app, via fake LLMfree, deterministic
contract/Schema stability: every artifact validates; JSON Schema export is goldenfree
invariants/The architectural claims abovefree
e2e/One real LLM run against a live browser@pytest.mark.live, opt-in

Offline by default

Everything except e2e/ runs with no API key and no network. 2,158 tests (1,800 of them edge cases: boundary, malformed, hostile and out-of-order input), mypy --strict clean. CI configures no API key on purpose — if a test needs the model, CI must fail rather than silently skip.

Continuous integration

.github/workflows/ci.yml runs the same targets a contributor runs locally, as separate steps so a failure names its stage:

  1. Lint and format, then strict type check
  2. Architectural invariants: the five import contracts, plus a check that every path the docs cite exists
  3. Secret scan (make secrets HISTORY=1) over the tree and every commit. It never prints what it finds, since echoing a secret into CI logs is itself a leak
  4. The offline test suite, the reviewer-path smoke test, and the full 14-stage demo

The workflow holds read-only permissions, and a newer push cancels a superseded run on the same ref. CI has no model key on purpose. The demo's evidence is uploaded as a build artifact, so a reviewer can download exactly what that run produced.

A second workflow, .github/workflows/verify.yml, runs the slow checks CI does not run on every push: the edge-case runs (make edge-cases), the test report with line and branch coverage, and the mutation check, which breaks each invariant on purpose and fails if any deliberate violation goes uncaught. It runs nightly, on demand, and on pushes that touch code or tests, and uploads evidence/tests/ and evidence/edge-cases/ as an artifact. Dependabot keeps the pinned toolchain and the Actions versions current, weekly and grouped.

The one thing CI cannot do on every commit is spend real tokens. A separate, manually triggered Live discovery workflow runs make discover against your gateway, then make verify-live, and uploads the evidence as a build artifact to review before anything is committed. It needs an OMNIROUTE_API_KEY secret, plus CUA_LLM_BASE_URL and CUA_LLM_MODEL variables, and the gateway must be reachable from a GitHub-hosted runner.

The fake LLM

agent/fake.py implements LlmPort by replaying a recorded transcript. This exercises the real discovery loop — same policy, same resolver, same evidence bus — for free.

Risk: fixture drift. Mitigated by regenerating fixtures from a live run via script, plus a contract test asserting fixture shape matches the live LlmPort schema.

Run a Discovery

Discovery is the probabilistic half: an LLM drives a live UI to a goal and we record what it did. It is the expensive path, run once per capability.

Prerequisites

cp .env.example .env     # set OMNIROUTE_API_KEY, CUA_LLM_BASE_URL, CUA_LLM_MODEL
make app                 # the mock back-office on :8811

Run

cua discover \
  --goal "Look up member 12345 and read their current savings balance" \
  --target http://localhost:8811 \
  --out evidence/capabilities/

What happens

  1. The driver observes the surface and normalizes it into a UiSnapshot
  2. The snapshot is rendered for the model — not raw HTML, and redacted on the way out
  3. The model emits a tool call from the closed action space. It cannot invent an action
  4. PolicyEngine authorizes it. Off-allowlist or irreversible actions are blocked
  5. The resolver resolves the target; the dispatcher dispatches; the bus records everything
  6. Loop until the goal is met or a stop condition fires
  7. On success, the compiler emits a draft Capability

Output

evidence/discovery/<run_id>/
  trace.jsonl        authorize / resolve / dispatch / observe, actor-tagged;
                     each llm_call carries the gateway's provider and request id
                     and the measured latency_ms;
                     run_end carries the stop reason and budget (steps, tokens, elapsed)
  snapshots/         UiSnapshot per step
  screenshots/       redacted
  run_record.json    timings, model, tokens_used -- summed from the llm_call events,
                     and equal to the budget's token count
evidence/capabilities/<id>@<version>.yaml

Verify it was live

make verify-live                                   # the newest evidence/discovery/disc-* run
make verify-live RUN=evidence/discovery/<run_id>  # a specific one

Hard checks fail the command: a named, non-scripted model, tokens spent, and a gateway-issued request id on every model call. A median latency under 100 ms only warns, because a hosted model rarely answers that fast but a local gateway with a warm cache can. Runs recorded before latency_ms existed fall back to timestamp gaps and are labelled derived, not measured.

If it fails

SymptomLook at
verify-live says the run was scriptedNo OMNIROUTE_API_KEY was set, so discovery fell back to the recorded transcript (model_name: fake/scripted)
Stops at max_stepsthe llm_call events in trace.jsonl — is it looping on one screen?
POLICY_DENIEDExpected if the goal needs an irreversible action
TARGET_AMBIGUOUSThe screen genuinely has two matching controls. Good — it refused.
Model can't find a controlCheck the UiSnapshot in snapshots/

Run a Replay

Replay is the deterministic half: the production execution path. No model, no decisions, no surprises.

make app
cua replay evidence/capabilities/corebank.member.savings_balance@1.0.0.yaml \
  --input member_id=67890

Exercising error paths

Faults are armed on the mock app out of band, never through cua replay (it has no fault flag): POST /_control/arm with {"fault": <name>, "count": <n>} (-1 = every request), after signing in. cua console --capability <ref> --arm-fault <name> does the same for a supervised run.

Input / armed faultResult
--input member_id=99999BUSINESS_OUTCOME(member_not_found)
transient_load, count 1RECOVERABLE → recovered → SUCCESS
transient_load, count -1RECOVERY_EXHAUSTED → escalates (NEEDS_HUMAN)
undeclared_dialogUNEXPECTED_STATE → fails closed
session_timeoutre-authenticated by run_capability corebank.auth.sign_on → SUCCESS; escalates if that cannot run

The session_timeout remedy needs corebank.auth.sign_on in the catalog and the operator credentials as secrets (CUA_SECRET_COREBANK_OPERATOR_ID, CUA_SECRET_COREBANK_OPERATOR_PASSWORD). Without either it fails closed as RECOVERY_EXHAUSTED.

Debugging a failure

FailureUsual cause
TARGET_NOT_FOUNDThe screen isn't what the artifact expects
TARGET_AMBIGUOUSDescriptor too loose, or genuinely two matches
CHECKPOINT_FAILEDSteps ran but goal state wasn't reached
UNEXPECTED_STATECapability doesn't declare this screen
RECOVERY_EXHAUSTEDThe "transient" condition wasn't transient

Determinism

Same artifact + same inputs + same app state ⇒ identical decision trace. If two replays diverge, that is a bug in this system, not flakiness.

Handle an Escalation

A run stopped because it could not safely proceed. This is a designed outcome, not a crash.

Taking control

cua console          # http://localhost:8812

Claim takes the lease. The lease epoch increments, and any in-flight automation dispatch is refused with LEASE_LOST.

You are driving the same live session, not a copy.

What you can and cannot do

You, during an intervention
Domain allowlistStill enforced
Ordinary actionsAllowed, recorded
Irreversible actionsAllowed with explicit confirmation, recorded
EvidenceEverything captured, tagged actor=HUMAN

Handing back

Release diffs session state, records human_delta, and returns the lease to automation. The console does not restart the run itself; when a caller resumes it (ReplayExecutor.resume(), as cua demo does), the executor:

  1. Scans forward from the step that escalated
  2. Skips a step only if its postcondition already holds (a step with no postcondition is never skipped)
  3. Resumes at the first step not visibly done, provided its precondition holds
  4. Otherwise stops with UNEXPECTED_STATE

Afterwards

A recurring escalation is a capability defect, not an operations problem. Usual fixes:

  • Screen seen repeatedly but undeclared → add an outcomes or recovery entry
  • TARGET_AMBIGUOUS on the same step → tighten the descriptor
  • Rising drift_score for one tenant → add a TenantBinding overlay, or re-record