Vercel Labs Agent Desktop Reads App UIs Through Accessibility Trees, Not Pixels
Summary
Vercel Labs publishes Agent Desktop, a Rust CLI and C library that reads desktop app UI through OS accessibility trees on macOS, Windows and Linux. Agents act on snapshot refs, then re-snapshot to verify, avoiding pixel matching. It ships 60 command names, 56 operational, and installs via npm.
Key Points
- Vercel Labs says key-down, key-up, mouse-down and mouse-up refuse to run until a daemon can release a held key or button.
- On macOS, a private command host reused per snapshot namespace and build exits after 15 idle minutes.
- A bare ref such as @e8 is rejected with INVALID_ARGS because refs belong to their snapshot.