All projects

Project 01 / Computer-use agents · Python

Hands

Lets an AI agent operate legacy back-office software that has no API: a model works out a task once, and production replays it deterministically — no model in the loop.

Private repository — happy to walk you through it

Year
2026
Role
Solo — design and build
Type
Computer-use automation
Status
Working end-to-end demo
Model calls on replay
0
Deterministic, typed results
Model it was built on
8B
Local Qwen3-VL, zero invalid decisions
Discovery on Sonnet 5.5
27s
7 steps, vs ~2 min locally
Live evidence cases
14
Outcomes, recoveries, handoff

01

The problem

Banks and credit unions run on back-office software with no API — framesets, layout tables, no stable ids. Automating it means brittle scripts that break on the next release, or an AI agent that improvises every click, pays for model calls on every run, and can't promise the same result twice.

Hands splits the job in two. A model explores the app once and records what it did as a typed, versioned capability — a function signature with inputs, outputs and an error vocabulary. Production calls that capability, and the engine replays it step by step with no model at all.

02

How it works

From one discovery to every replay
  1. 01

    Discover

    The model sees the accessibility tree and a screenshot and picks one action. The loop validates it, re-prompts on bad output, catches stuck loops and substitutes secrets.

  2. 02

    Record

    Each action becomes a step with ranked locators — role and name, label, text anchor, structural path, then a visual offset — each kept only if it resolves uniquely.

  3. 03

    Capability

    A versioned contract: typed inputs and outputs, business outcomes, bounded recoveries, provenance. An agent sees it as a tool definition.

  4. 04

    Replay

    For every step: wait for the precondition, run monitors, check policy, act, wait for the postcondition. No fixed sleeps, no model.

  5. 05

    Typed result

    Success with outputs, a business outcome like MEMBER_NOT_FOUND, an escalation, or a hard failure naming the step, what was expected and what was seen.

  6. 06

    Human handoff

    When a run is stuck, an operator takes over the same live session through a streamed view, then hands it back and the run resumes.

03

Key decisions

  • See through the accessibility tree, not the DOM

    Browsers and desktop operating systems expose the same tree, so a recorded flow never contains CSS, XPath or a toolkit name — and one engine drives both a web app and a macOS app.

    Ruled outCSS and XPath selectors, or screenshot coordinates alone

  • Build against a weak model

    All development used a local 8B vision model. The loop owns validation, retries and stuck detection, so a stronger model only makes it better.

  • Guardrails at the seam, not in the prompt

    Every action — from the model, the engine or a human — passes a deny-by-default policy and a control lease before it reaches the app. Risky actions like Submit or Transfer never run during discovery.

    Ruled outTelling the model what not to click

  • Runtime knowledge is data

    "Member not found", session expiry, maintenance notices: outcomes and recoveries live in a per-vendor profile, so the engine has no app-specific code.

  • Tenants are overlays, not forks

    A second institution on the same product is a small overlay — renamed labels, one re-targeted table cell — with the step list unchanged. Fallback matches flag drift early.

  • Secrets never reach the model

    Credentials resolve at the last moment from placeholders. Every write passes one redactor that masks secrets, checksum-valid card and routing numbers, and sensitive screen regions.

04

In the code

Discover once with a model, then replay with none — every condition comes back typed
# Discover once — a model explores the app and records a capability
hands discover "Look up member {{member_number}} in CoreOne and \
  return the balance of their savings share as savings_balance." \
  -i member_number=10001

# Replay forever — no model in the loop, typed results back
hands replay coreone.member_balance -i member_number=10002
# → success: savings_balance

hands replay coreone.member_balance -i member_number=99999
# → business_outcome: MEMBER_NOT_FOUND

hands replay coreone.member_balance -i member_number=abc
# → invalid_input — rejected before the app is touched

# Same capability, second institution — a tenant overlay, not a fork
hands replay coreone.member_balance -i member_number=10001 --tenant prairie

05

What I'd do next

  • Learn the unhappy paths. Discovery records the happy path; outcomes and recoveries come from a hand-written vendor profile. Next: a bounded, policy-checked model fallback when replay fails, proposing new profile rules for review.

  • Drift baselines. Drift is detected from fallback matches today. Storing per-step structural fingerprints would catch it sooner.

  • A real operator console. It's one local page with one session and no authentication. Production needs operator identity, a queue and a remote session broker — the lease model stays the same.

  • Windows. A UI Automation surface is one more adapter, the same shape as the macOS one.

06

Stack

Core
Python 3.11Pydanticuv
Surfaces
Playwright (CDP)macOS Accessibility API
Models
Claude Sonnet 5.5Qwen3-VL 8B via Ollama
Handoff
CDP screencastInput injection
Test
pytestRecorded model cassettes