Project 01 / Computer-use agents · Python
Hands
Lets an AI agent operate legacy back-office software that has no API: a model works out a task once, and production replays it deterministically — no model in the loop.
Private repository — happy to walk you through it
- Year
- 2026
- Role
- Solo — design and build
- Type
- Computer-use automation
- Status
- Working end-to-end demo
- Model calls on replay
- 0
- Deterministic, typed results
- Model it was built on
- 8B
- Local Qwen3-VL, zero invalid decisions
- Discovery on Sonnet 5.5
- 27s
- 7 steps, vs ~2 min locally
- Live evidence cases
- 14
- Outcomes, recoveries, handoff
01
The problem
Banks and credit unions run on back-office software with no API — framesets, layout tables, no stable ids. Automating it means brittle scripts that break on the next release, or an AI agent that improvises every click, pays for model calls on every run, and can't promise the same result twice.
Hands splits the job in two. A model explores the app once and records what it did as a typed, versioned capability — a function signature with inputs, outputs and an error vocabulary. Production calls that capability, and the engine replays it step by step with no model at all.
02
How it works
- 01
Discover
The model sees the accessibility tree and a screenshot and picks one action. The loop validates it, re-prompts on bad output, catches stuck loops and substitutes secrets.
- 02
Record
Each action becomes a step with ranked locators — role and name, label, text anchor, structural path, then a visual offset — each kept only if it resolves uniquely.
- 03
Capability
A versioned contract: typed inputs and outputs, business outcomes, bounded recoveries, provenance. An agent sees it as a tool definition.
- 04
Replay
For every step: wait for the precondition, run monitors, check policy, act, wait for the postcondition. No fixed sleeps, no model.
- 05
Typed result
Success with outputs, a business outcome like MEMBER_NOT_FOUND, an escalation, or a hard failure naming the step, what was expected and what was seen.
- 06
Human handoff
When a run is stuck, an operator takes over the same live session through a streamed view, then hands it back and the run resumes.
03
Key decisions
See through the accessibility tree, not the DOM
Browsers and desktop operating systems expose the same tree, so a recorded flow never contains CSS, XPath or a toolkit name — and one engine drives both a web app and a macOS app.
Ruled outCSS and XPath selectors, or screenshot coordinates alone
Build against a weak model
All development used a local 8B vision model. The loop owns validation, retries and stuck detection, so a stronger model only makes it better.
Guardrails at the seam, not in the prompt
Every action — from the model, the engine or a human — passes a deny-by-default policy and a control lease before it reaches the app. Risky actions like Submit or Transfer never run during discovery.
Ruled outTelling the model what not to click
Runtime knowledge is data
"Member not found", session expiry, maintenance notices: outcomes and recoveries live in a per-vendor profile, so the engine has no app-specific code.
Tenants are overlays, not forks
A second institution on the same product is a small overlay — renamed labels, one re-targeted table cell — with the step list unchanged. Fallback matches flag drift early.
Secrets never reach the model
Credentials resolve at the last moment from placeholders. Every write passes one redactor that masks secrets, checksum-valid card and routing numbers, and sensitive screen regions.
04
In the code
# Discover once — a model explores the app and records a capability
hands discover "Look up member {{member_number}} in CoreOne and \
return the balance of their savings share as savings_balance." \
-i member_number=10001
# Replay forever — no model in the loop, typed results back
hands replay coreone.member_balance -i member_number=10002
# → success: savings_balance
hands replay coreone.member_balance -i member_number=99999
# → business_outcome: MEMBER_NOT_FOUND
hands replay coreone.member_balance -i member_number=abc
# → invalid_input — rejected before the app is touched
# Same capability, second institution — a tenant overlay, not a fork
hands replay coreone.member_balance -i member_number=10001 --tenant prairie05
What I'd do next
Learn the unhappy paths. Discovery records the happy path; outcomes and recoveries come from a hand-written vendor profile. Next: a bounded, policy-checked model fallback when replay fails, proposing new profile rules for review.
Drift baselines. Drift is detected from fallback matches today. Storing per-step structural fingerprints would catch it sooner.
A real operator console. It's one local page with one session and no authentication. Production needs operator identity, a queue and a remote session broker — the lease model stays the same.
Windows. A UI Automation surface is one more adapter, the same shape as the macOS one.
06
Stack
- Core
- Python 3.11Pydanticuv
- Surfaces
- Playwright (CDP)macOS Accessibility API
- Models
- Claude Sonnet 5.5Qwen3-VL 8B via Ollama
- Handoff
- CDP screencastInput injection
- Test
- pytestRecorded model cassettes