How to choose a GUI-agent benchmark
Choose a benchmark by the capability and environment you need to measure. Screenshot target localization, live website navigation, Android workflows, and cross-application desktop tasks answer different questions.
| Benchmark | Environment | Primary focus |
|---|---|---|
| OSWorld | Real desktop environments | Task outcomes checked through execution and application/system state. |
| WebArena | Reproducible websites | Functional completion of web tasks. |
| AndroidWorld | Android applications | Parameterized interactive tasks with initialization and success checks. |
| ScreenSpot-Pro | Professional interface screenshots | Precise localization of requested GUI targets. |
Compare source-reviewed benchmark protocols, task populations, and interfaces.
Which benchmark is best?
There is no single answer without a deployment target. For a desktop assistant, start with desktop task execution. For a browser agent, use web tasks and the relevant site conditions. For mobile control, test the mobile environment. For a grounding model, isolate target localization before interpreting end-to-end task outcomes. This recommendation follows from the different tasks defined by the linked benchmark papers.
What must be recorded with a score?
- The benchmark version or commit and task split.
- The exact model checkpoint, agent harness, tools, and observation interface.
- The action or step budget and any retries or multiple attempts.
- The evaluator, success criterion, and aggregation method.
- The environment setup, run date, and whether sites or apps changed.
- Whether the result was reproduced independently or reported by the publisher.
Without these details, apparently similar percentages may refer to different experiments. A static grounding accuracy and a completed-task rate should not be placed in a shared ranking.
How do I check reproducibility?
Follow the benchmark’s current official setup and evaluation instructions. Compare them with the version used in the paper. The current OSWorld repository, WebArena repository, and AndroidWorld repository are starting points; availability of code alone does not establish a successful reproduction.
Where can I find more benchmarks?
Browse the reviewed benchmark directory or use the interactive benchmark filter. Each resource exposes its primary evidence and any recorded specifications.
Editorial update: 2026-10-16. The comparison is about evaluation scope, not a claim that one benchmark or model is universally superior.