# Computer-Use Automation System (CUA) — Full Technical Specification & Documentation > Core thesis: an LLM is allowed to be uncertain exactly once; the run it produces is > compiled into a typed artifact that replays with no model in the loop. > Canonical URL: https://adityamhaske.github.io/computer-use-automation/ > GitHub: https://github.com/adityamhaske/computer-use-automation --- ## 1. Overview CUA is an enterprise computer-use automation system engineered for regulated financial back-offices (banks and credit unions). An LLM is permitted to explore an arbitrary legacy application with no API exactly once. That exploration trace is compiled into an immutable, typed, content-hashed Capability Artifact. All subsequent runs execute deterministically with zero model calls in the loop. Built for the back-office applications banks and credit unions run when there is no API and the only way in is to drive the UI. Every action is authorized at one chokepoint and recorded on an evidence bus; every sink -- logs, artifacts, snapshots, screenshots and outbound model prompts -- passes through the redactor. The limits of that are stated in REPORT.md section 6: free-text redaction is patterns plus registered literals, so a member name is protected only where a capability declares the field sensitive. ``` Goal + Target ──► LLM Discovery Run ──► Capability Artifact ──► Deterministic Replay ──► Typed Outputs (once) (reviewed, sealed) (every time after) │ stuck? ───────┴──► Human takes the live session, acts, hands it back ``` --- ## 2. Quick Start & Prerequisites ### Prerequisites - Python 3.11 or later - Playwright Chromium (`make setup` installs this) - macOS or Linux - An optional OmniRoute API key (`OMNIROUTE_API_KEY`, with `CUA_LLM_BASE_URL` and `CUA_LLM_MODEL`) for live LLM discovery runs; `OPENROUTER_API_KEY` is still read as a fallback. Offline recordings are provided for zero-API-key testing ### Installation ```bash git clone https://github.com/adityamhaske/computer-use-automation.git cd computer-use-automation make setup # Creates venv, installs dependencies, installs Chromium ``` ### Running the Demo ```bash ./start.sh # Runs the 14-stage verified execution story end to end ``` ### Testing ```bash make check # lint + strict typecheck + invariants + secret scan + tests make test # the offline suite -- no API key, no network make secrets # fail on any key-shaped string in tracked files (HISTORY=1: every commit) make discover # live LLM discovery; needs OMNIROUTE_API_KEY make verify-live # check the newest discovery run was a live model, not a script make edge-cases # 22 hostile/boundary scenarios run for real; every run saved to evidence/edge-cases/ make test-report # whole suite under line + branch coverage -> evidence/tests/ make mutation-check # break each invariant on purpose; prove the suite goes red ``` --- ## 3. The 14-Stage Verified Execution Story The demo executes 14 deterministic stages against a live back-office application: 1. Mock Back-Office Running: Hostile multi-frame banking application starts on port 8811. 2. LLM Discovery: Model discovers navigation path to goal once. 3. Capability Compiled: Trace compiled into typed, frozen, content-hashed artifact. 4. Deterministic Replay: New member inputs executed with 0 models in the loop. 5. Business Outcome: Member not found handled as a legitimate answer, not a crash. 6. Injected 502 Fault: Declared recovery rule catches HTTP 502, remediates, and retries. 7. Fault Past Budget: Exceeded retry budget triggers clean handoff to human operator. 8. Malformed Input: Input schema guard rejects invalid inputs before touching the browser. 9. Undeclared Screen: Unrecognized screen state fails closed (`UNEXPECTED_STATE`). 10. Human Takeover: Operator claims session with exclusive Lease and drives screen. 11. Re-Anchor Resume: Engine reconciles screen state and resumes without duplicate mutation. 12. Agent Tool Call: Capability exposed to caller agents as a standard typed function. 13. Approval Gate: Unattended replay is gated on a human review of the evidence, not just content-hash integrity (`cua catalog approve --from-eval`). 14. Generated Test: The same artifact emits a runnable pytest + Playwright test, using only role/accessible-name locators -- no CSS -- so it outlives this system (`cua codegen`). --- ## 4. System Architecture & Package Map ``` Action ──► TargetResolver ──► PolicyEngine ──► SurfaceDriver (resolve or refuse) (authorize) (dispatch) ``` | Package | Purpose & Ownership | Permitted Imports | |---|---|---| | `cua/domain` | Types only: `Capability`, `TargetDescriptor`, `Action`, `UiSnapshot`, `RunRecord`, taxonomy | Nothing from `cua` (pure) | | `cua/perception` | Normalizing raw surface observation into `UiSnapshot`; fingerprinting | `domain` | | `cua/targeting` | The resolution ladder, scoring, ambiguity, drift | `domain`, `perception` | | `cua/policy` | Allowlist, risk tiers, redaction, secrets, `AuthorizedAction` | `domain`, `targeting` | | `cua/hitl` | `Lease`, `SessionBroker`, intervention queue, operator console | `domain`, `policy` | | `cua/runtime` | `dispatcher.py` — the sole importer of `surfaces` | everything below | | `cua/surfaces` | `SurfaceDriver` port; Playwright/CDP driver; desktop stub | `domain`, `perception` | | `cua/agent` | `LlmPort`, prompts, the discovery loop | `runtime` and below | | `cua/replay` | The deterministic executor — **never imports agent** | `runtime` and below | | `cua/recorder` | Compiling a discovery trace into a `Capability` | `domain`, `targeting` | | `cua/evidence` | Structured, redacted, actor-tagged run records | `domain`, `policy` | | `cua/catalog` | Capability store, tenant overlays, tool-spec export | `domain` | | `cua/cli` | Unified CLI entry points | everything | --- ## 5. The Ten Architectural Invariants ### 1. Replay never calls or imports the LLM `cua.replay` may not import `cua.agent`, `httpx`, or any model client — directly or transitively. - Why: It is the core thesis. Replay must be deterministic, auditable, and cost-free. - Enforced by: `.importlinter` contract `no-llm-in-replay`, `tests/invariants/test_import_contracts.py`. ### 2. No action reaches a driver without policy authorization The only path to a surface is: `Action -> TargetResolver -> PolicyEngine -> SurfaceDriver`. `cua.runtime.dispatcher` is the only module permitted to call `dispatch()`. - Why: Guardrails cannot be bypassed. - Enforced by: `policy-chokepoint` contract and `tests/invariants/test_policy_chokepoint.py`. ### 3. Target identity is a semantic TargetDescriptor, never a raw CSS selector Artifacts describe controls by role, accessible name, scope, structural anchor, and ordinal. - Why: Survives restyling, re-branding, and cross-platform execution (Playwright, Windows UIA). - Enforced by: `surface-neutral-targeting` contract and `tests/integration/test_variant_b.py`. ### 4. Human input cannot bypass policy or evidence The operator console never injects directly into the page. It submits `raw_input` actions to `SessionBroker`, which routes them through the policy chokepoint under the `HUMAN` policy profile. A raw click is classified by the control it lands on (hit-tested against node geometry; a key by the focused control), so an irreversible control needs the operator's explicit confirmation. The console refuses WebSocket and state-changing requests whose `Origin` is not its own. - Enforced by: `tests/invariants/test_human_control_safety.py`. ### 5. Unknown states fail closed If the observed UI matches no declared precondition, checkpoint, outcome, or recovery rule, it is `UNEXPECTED_STATE` and execution immediately halts. If a target is ambiguous, the resolver refuses with `TARGET_AMBIGUOUS`. - Enforced by: `tests/integration/test_fault_matrix.py`. ### 6. Every sink is redacted Logs, artifacts, evidence files, screenshots, and outbound LLM prompts are redacted. Secrets are `{$secret: ref}` references resolved only at dispatch time. - Enforced by: `tests/invariants/test_redaction.py`. ### 7. Business outcomes are not failures Observable conditions are strictly partitioned into: - `BUSINESS_OUTCOME`: Real answer (`member_not_found`, `account_closed`) -> terminal success (Exit 0). - `RECOVERABLE`: Bounded transient condition (`transient_load`, `session_expired`) -> retry. - `HARD_FAILURE`: Defect (`target_not_found`, `checkpoint_failed`) -> terminal failure. - `UNEXPECTED_STATE`: Unrecognized state -> fail closed to human. - Enforced by: `tests/integration/test_fault_matrix.py`. ### 8. Automation may only act while it holds the session Every automation dispatch checks: who holds this session, and has it changed hands since this run was authorized (`expected_epoch`). Either can refuse with `LEASE_LOST`. - Enforced by: `tests/integration/test_handoff.py`. ### 9. Capabilities are immutable A capability artifact at `id@version` is frozen and SHA-256 content-hashed. - Enforced by: `tests/contract/test_capability_schema.py`. ### 10. domain/ is pure Zero I/O, zero network, zero browser, zero clock. - Enforced by: `.importlinter` contract `pure-domain`. --- ## 6. Capability Schema Reference A capability document contains: - `id`: Unique identifier (e.g. `memberdesk.savings_balance`) - `version`: SemVer string (e.g. `1.0.0`) - `content_hash`: SHA-256 hash of all fields excluding the hash itself - `inputs`: JSON Schema defining required caller arguments - `outputs`: JSON Schema defining returned values - `steps`: Ordered list of deterministic steps: - `action`: `click`, `fill`, `select_option`, `press_key` - `target`: `TargetDescriptor` (role, name, anchor, hints) - `checkpoint`: Expected post-action state assertion - `outcomes`: Declared business outcomes and their extractors - `recovery`: Declared retry rules for recoverable transients - `sensitive_names`: Fields to redact from all audit sinks --- ## 7. Target Resolution Ladder ``` Rung 1: semantic_exact role + accessible name, exact, in scope Rung 2: semantic_normalized as above, case/whitespace/punctuation folded Rung 3: structural_anchor position relative to nearby label text Rung 4: hint_cached stored CSS hint, accepted only if it also satisfies role/name Rung 5: ordinal_in_region Nth control of a role within a scope (vision: not implemented; never used in replay) ``` The first rung that yields any candidate decides: exactly one wins; more than one is refused with TARGET_AMBIGUOUS unless the descriptor carries an explicit `ordinal`; none at any rung is TARGET_NOT_FOUND. There is no ambiguity threshold and no candidate scoring. Drift: resolving at a weaker rung than the one recorded at discovery. `drift_score` is the fraction of a run's resolutions that drifted (0.0-1.0). It is a signal only -- no threshold on it stops a run. --- ## 8. Error Taxonomy & Fault Matrix | Observable State | Classification / code | Action Taken | RunStatus (exit) | |---|---|---|---| | member record found | `EXPECTED` | Extract outputs, verify checkpoint | `SUCCESS` (0) | | `member_not_found` | `BUSINESS_OUTCOME` | Return typed outcome, write record | `BUSINESS_OUTCOME` (0) | | HTTP 502 (`transient_load`) | `RECOVERABLE` | Declared remedy (reload), retry step | resumes | | session expired (`session_timeout`) | `RECOVERABLE` | `run_capability corebank.auth.sign_on`, retry step | resumes | | recovery past `max_attempts` | `RECOVERY_EXHAUSTED` | Escalate if listed in `escalation.triggers` | `NEEDS_HUMAN` (2) | | target not found / ambiguous | `TARGET_NOT_FOUND` / `TARGET_AMBIGUOUS` | Escalate if listed in `escalation.triggers` | `NEEDS_HUMAN` (2) or `FAILED` (1) | | undeclared screen | `UNEXPECTED_STATE` | Stop; escalate if listed | `NEEDS_HUMAN` (2) or `FAILED` (1) | | operator holds / took the session | `LEASE_LOST` | Refuse the dispatch | `FAILED` (1) | A debuggable defect is `FAILED` carrying a `FailureCode`; there is no separate hard-failure class. --- ## 9. CLI Reference ### `cua discover` ```bash cua discover --goal "Look up member balance" --target http://localhost:8811 --out evidence/capabilities/ ``` Options: - `--goal`: Natural language description of workflow - `--target`: Base URL of target application surface - `--out`: Directory where compiled capability artifact is written ### `cua replay` ```bash cua replay evidence/capabilities/memberdesk.savings_balance@1.0.0.yaml --input member_id=12345 ``` Options: - `--input`: Key-value pairs matching capability inputs schema - `--base-url`: Override base URL for target environment - `--no-headless`: Run with visible browser window for debugging - `--allow-irreversible`: Caller opt-in for an irreversible capability (an approval of its exact content is also required) - Faults are not a `cua replay` flag: they are armed on the mock app via `POST /_control/arm` (`transient_load`, `session_timeout`, `undeclared_dialog`, `validation_error`) ### `cua console` ```bash cua console --target http://localhost:8811 --port 8812 ``` Options: - `--target`: Application surface to supervise - `--port`: Port for web operator console (default 8812) --- ## 10. Summary for AI Agents & Tools When integrating CUA into an agent loop: - Invariants 1 and 2 guarantee that invoking a capability artifact will never spawn an uncontrolled LLM call or bypass policy chokepoints. - Outputs are fully typed and validated against the capability's declared schema. - All secrets are injected as `{$secret: ref}` references resolved at dispatch, ensuring zero credentials are leakable to calling agents or output logs.