What are computer-use agents?

A computer-use agent is an AI system that observes a software interface, decides what action to take, executes it, and checks the resulting state to work toward a user’s goal. The interface may be a browser, desktop application, or mobile device.

Some systems act mainly through screenshots, clicks, and keystrokes. Others combine graphical interaction with code and tool calls. These are different execution designs, not interchangeable claims of capability. See the primary descriptions of UI-TARS and OpenGCUA.

How does a computer-use agent work?

1. Observe: obtain screenshots, page structure, accessibility information, or other exposed state.
2. Plan and ground: decide a useful next step and identify the interface element or tool that can perform it.
3. Act: click, type, scroll, invoke a tool, or execute code within the system’s permitted interface.
4. Check: inspect the new state, determine progress, and retry, replan, stop, or ask for help.

This is a practical decomposition rather than a single mandated architecture. Agent S is one example that combines planning with retrieved experience and a computer interaction interface.

Are GUI grounding and task completion the same?

No. Grounding asks whether a model can locate the correct target on a screen. Task completion additionally involves choosing the right actions, handling transitions, tracking progress, and verifying the final result. A strong grounding score alone does not establish that an agent can finish a long workflow.

ScreenSpot-Pro evaluates grounding in professional interfaces, while OSWorld evaluates tasks in real computer environments. Use the benchmark appropriate to the capability you want to measure.

Which resources should I start with?

• For desktop evaluation: benchmarks and the OSWorld paper.
• For model behavior and perception: GUI model families and checkpoints.
• For training: training data and supervision types.
• For deployment: frameworks and execution environments.
• For the literature: reviewed papers and surveys.

How should I interpret this catalog?

A repository, a paper, a dataset, and a model checkpoint can describe related parts of one project. Their counts are not independent project counts. The editorial policy distinguishes primary-source review from metadata verification and independent reproduction.

Editorial update: 2026-10-16. This guide synthesizes the linked primary research; it does not independently validate experimental claims.