I built the Playwright for desktop apps. 80% token savings
Key takeaways
- It gives structured access to any application through OS accessibility trees — no screenshots, no pixel matching, no browser required.
- Grant it in System Settings > Privacy & Security > Accessibility by adding your terminal app, or:
- agent-desktop permissions --request # trigger system dialog Language bindings (FFI) Every GitHub Release ships a prebuilt C-ABI cdylib alongside the CLI tarballs.
It gives structured access to any application through OS accessibility trees — no screenshots, no pixel matching, no browser required.
Key Features Native Rust CLI: Fast, single binary, no runtime dependencies C-ABI cdylib (libagent_desktop_ffi): Load once from Python / Swift / Go / Ruby / Node / C instead of forking the CLI per call 53 commands: Observation, interaction, keyboard, mouse, notifications, clipboard, window management Progressive skeleton traversal: 78–96% token reduction on dense apps via shallow overview + targeted drill-down Snapshot & refs: AI-optimized workflow using deterministic element references (@e1, @e2) AX-first interactions: Every action exhausts pure accessibility API strategies before falling back to mouse events Structured JSON output: Machine-readable responses with error codes and recovery hints Works with any app: Finder, Safari, System Settings, Xcode, Slack — anything with an accessibility tree Installation npm (recommended) npm install -g agent-desktop # downloads prebuilt binary automatically Or without installing:
npx agent-desktop snapshot --app Finder -i From source git clone https://github.com/lahfir/agent-desktop cd agent-desktop cargo build --release cp target/release/agent-desktop /usr/local/bin/ Requires Rust 1.78+ and macOS 13.0+.