AwesomeCUAGitHub ↗
RESOURCE CATEGORY

Computer-use agent benchmarks

Browse 89 reviewed benchmarks connected to computer-use, GUI, web, and mobile agents. Each entry links to primary evidence and records its scope.

Review means AI-assisted editorial inspection, not independent reproduction. Unknown licenses and specifications remain unknown.

Compare source-reviewed specifications

Benchmarks · curated

Adversarial Web Agents

Evaluation resources for adversarial attacks on multimodal web agents.

Release date not recorded · web
Benchmarks · curated

AgentBench

Agent evaluation suite with containerized tasks and a function-calling interface, including operating-system and shopping environments.

Release date not recorded · cross-platform, web
Benchmarks · curated

AgentDAM

Web-agent benchmark for measuring unnecessary use of sensitive personal information during end-to-end task execution.

Release date not recorded · web
Benchmarks · curated

AgentRewardBench

A benchmark and evaluation toolkit for comparing automatic web-agent judges against expert judgments of task success, side effects and repetition.

Release date not recorded · web
Benchmarks · curated

AndroidArena

Android environment for evaluating language-model-driven agents.

Release date not recorded · android
Benchmarks · curated

AndroidLab

Reproducible Android benchmark of 138 tasks across nine apps on predefined virtual devices, run via AVD on Mac or Docker on Linux, released with the Android Instruct training set.

Release date not recorded · android
Benchmarks · curated

AndroidWorld

Android environment with programmatic task evaluation for mobile agents.

Release date not recorded · android
Benchmarks · curated

AppSim-Bench

Simulated mobile applications and evaluation tooling for reproducible GUI-agent tasks with controlled application state.

Release date not recorded · android
Benchmarks · curated

AppWorld

Simulated applications, user data, and state-based task checks for evaluating agents that interact through APIs and code.

Release date not recorded · unspecified
Benchmarks · curated

B-MoCA

Mobile control benchmark with varied interface configurations.

Release date not recorded · android
Benchmarks · curated

BrowserART

Red-teaming benchmark and test suite from Scale AI evaluating whether refusal-trained language models can be jailbroken when deployed as browser agents.

Release date not recorded · web
Benchmarks · curated

BrowserGym

Unified interfaces for browser-agent tasks and benchmark environments.

Release date not recorded · web
Benchmarks · curated

ColorBench

A graph-based mobile GUI benchmark that evaluates long tasks through screenshot states and multiple valid or erroneous transitions, including reflection, replanning and backtracking.

Release date not recorded · android
Benchmarks · curated

CRAB

Cross-environment benchmark framework for multimodal agents.

Release date not recorded · desktop, android
Benchmarks · curated

CUAVerifierBench

Human-annotated browser-agent trajectories for comparing automated task-success and process-quality verifiers against reviewer judgments.

Release date not recorded · web
Benchmarks · curated

EIA

Evaluation suite for Environmental Injection Attacks that embed invisible malicious HTML elements into benign websites to test privacy leakage in generalist web agents.

Release date not recorded · web
Benchmarks · curated

FedMABench

A federated mobile-agent training and evaluation suite that varies user-data heterogeneity and application distribution, with released partition annotations and inference/scoring scripts.

Release date not recorded · android
Benchmarks · curated

GUI-Robust

Evaluation scripts and linked data for comparing GUI action predictions and task execution under ordinary and anomalous interface conditions.

Release date not recorded · cross-platform
Benchmarks · curated

GUIHalluBench

Measures hallucination in UI-element parsing by matching the boxes and descriptions a model lists for 1,000 English and 1,000 Chinese screenshots against human annotations.

Release date not recorded · cross-platform
Benchmarks · curated

iOSWorld

Native iPhone-simulator benchmark with shared user data, application workflows, agent runners, and rubric-based scoring.

Release date not recorded · ios
Benchmarks · curated

LlamaTouch

Mobile UI task evaluation using application state information.

Release date not recorded · android
Benchmarks · curated

MacAgentBench

676 macOS tasks across 25 applications, each run in its own Docker container and scored by deterministic rule-based checks with multiple checkpoints.

Release date not recorded · desktop, macos
Benchmarks · curated

MCPWorld

Evaluation environment combining computer interaction with tool access.

Release date not recorded · cross-platform
Benchmarks · curated

Mind2Web

Human demonstrations and tasks for generalist web navigation research.

Release date not recorded · web
Benchmarks · curated

Mind2Web 2

Long-horizon agentic web search benchmark whose Agent-as-a-Judge framework scores time-varying, citation-backed answers, with evaluation scripts for its dev and test sets.

Release date not recorded · web
Benchmarks · curated

MiniWoB++

Over 100 web-interaction environments extending OpenAI's MiniWoB, controlled from Python through a Gymnasium API that drives the browser with Selenium.

Release date not recorded · web
Benchmarks · curated

MLA-Trust

Unified evaluation framework measuring truthfulness, controllability, safety, and privacy of multimodal LLM agents across interactive web and mobile GUI tasks.

Release date not recorded · web, android, cross-platform
Benchmarks · curated

MMBench-GUI

Four-level GUI-agent benchmark across six platforms, from content understanding and element grounding to task automation and collaboration, scored with an efficiency-quality area metric.

Release date not recorded · cross-platform
Benchmarks · curated

MMInA

Multimodal web tasks in a WebArena-based environment that chain from 1 to 10 hops across sites such as a shopping store and Wikipedia, testing long-chain reasoning.

Release date not recorded · web
Benchmarks · curated

MobBench task definitions

A public set of real-device mobile task definitions used to evaluate MobA, with Chinese and English task material packaged in a versioned JSON release.

Release date not recorded · android
Benchmarks · curated

Mobile-Bench-v2 released data

Offline mobile-agent task data spanning common tasks and released noisy, contaminated and ambiguous-instruction subsets.

Release date not recorded · android
Benchmarks · curated

Mobile-Eval-E task definitions

Task definitions for Mobile-Agent-E’s long-horizon, multi-app Android evaluation, packaged as a 25-record test split with rubrics and human-reference operation strings.

Release date not recorded · android
Benchmarks · curated

MobileAgentBench

Automated benchmark for mobile LLM agents whose default tasks use ten Simple Mobile Tools Android apps, installed as a Python library that detects task completion.

Release date not recorded · android
Benchmarks · curated

MobileGUI-VBench

Mobile-app task definitions covering everyday scenarios, with annotated preconditions and clarification requirements for evaluating agent behavior.

Release date not recorded · unspecified
Benchmarks · curated

MobileGym

A browser-hosted mobile-app simulator with programmable state, parameterized tasks and deterministic state-checking judges for parallel GUI-agent evaluation and online reinforcement learning.

Release date not recorded · web
Benchmarks · curated

MobileGym++ / MobileGym++ Bench

Browser-rendered mobile simulation with resettable app state and a shared task suite for GUI-only and GUI-plus-tool agents.

Release date not recorded · web
Benchmarks · curated

MobileSafetyBench

Android emulator testbed that scores mobile device-control agents on both safety and helpfulness, using ADB and Appium for observations, actions and success checks.

Release date not recorded · android
Benchmarks · curated

MobileWorld (Tongyi-MAI)

A containerized Android evaluation environment with GUI-only, agent-user interaction and MCP-augmented tasks, plus checks on application and backend state.

Release date not recorded · android
Benchmarks · curated

MobileWorldBench

Benchmark of semantic world modeling for mobile agents, with question-answering tasks and GPT-judged generation tasks, plus training data built on AitW and AndroidControl screens.

Release date not recorded · android
Benchmarks · curated

OmniAct

Dataset and benchmark for generating computer interaction programs.

Release date not recorded · desktop, web
Benchmarks · curated

Online Mind2Web

300 tasks on 136 live websites for evaluating web agents online, with outdated tasks replaced over time, human-reviewed submissions and the WebJudge automatic evaluator.

Release date not recorded · web
Benchmarks · curated

OpenApps

Configurable browser apps and task rewards for testing computer-use agents under changes in content, layout and appearance.

Release date not recorded · web
Benchmarks · curated

OS-Critic Bench

A cross-platform GUI-critic benchmark with mobile, desktop and web JSONL splits and screenshot assets, accompanied by public inference and accuracy-scoring scripts.

Release date not recorded · cross-platform, desktop, web
Benchmarks · curated

OS-Harm

Benchmark for safety of computer-use agents across misuse, prompt injection, and unintended model behavior.

Release date not recorded · desktop
Benchmarks · curated

OS-Map

An OSWorld-derived desktop benchmark with an Ubuntu virtual-machine environment and expanded task configurations for measuring breadth and depth of computer interaction.

Release date not recorded · desktop, linux
Benchmarks · curated

OSWorld

Execution-based evaluation of agents performing tasks in real desktop environments.

Release date not recorded · desktop, linux, windows, macos
Benchmarks · curated

OSWorld-G

Computer-use grounding benchmark with original and refined instructions, released with the Jedi synthetic data pipeline, a 4-million-example Jedi dataset and Jedi-3B and Jedi-7B models.

Release date not recorded · desktop
Benchmarks · curated

OSWorld-Human

Human reference trajectories and scoring code for measuring the action efficiency of computer-use agents on OSWorld tasks.

Release date not recorded · desktop
Benchmarks · curated

PopupAttack

Resources for evaluating GUI agents under malicious popup instructions.

Release date not recorded · web
Benchmarks · curated

PPTC

PowerPoint creation and editing benchmark with API-based execution and checks of the resulting presentation files.

Release date not recorded · unspecified
Benchmarks · curated

Precision CUA Bench

Evaluation resources for pixel-precise cursor grounding in code editors.

Release date not recorded · cross-platform
Benchmarks · curated

RecreationBench

Tasks for recreating reference applications with hybrid computer interaction.

Release date not recorded · cross-platform
Benchmarks · curated

RiOSWorld

Measures risky behavior of multimodal computer-use agents in an Ubuntu virtual machine, with risk scenarios such as phishing emails and released evaluation trajectories.

Release date not recorded · desktop, linux
Benchmarks · curated

SaaS-Bench

106 browser-driven business workflows across 23 self-hosted SaaS apps in six domains, scored by per-task verifiers that read each app's database and give partial credit.

Release date not recorded · web
Benchmarks · curated

SafeArena

Web-agent benchmark pairing benign and harmful tasks to study deliberate misuse risks.

Release date not recorded · web
Benchmarks · curated

ScienceBoard

Scientific-workflow tasks for multimodal agents in desktop software such as ChimeraX, KAlgebra, Celestia, Grass GIS and Lean 4, with apps modified to expose state for evaluation.

Release date not recorded · desktop
Benchmarks · curated

Screen Point-and-Read / Tree-of-Lens

Screen-reading benchmark and Tree-of-Lens implementation for interpreting user-indicated regions and their layout context.

Release date not recorded · cross-platform
Benchmarks · curated

ScreenSpot

GUI grounding test set of 1,272 screenshot instructions from iOS, Android, macOS, Windows and web interfaces, each labeled with the target element's bounding box.

Release date not recorded · cross-platform
Benchmarks · curated

ScreenSpot Pro

GUI grounding benchmark focused on professional high-resolution interfaces.

Release date not recorded · desktop, windows, macos, linux
Benchmarks · curated

ScreenSpot v2

Second version of the ScreenSpot grounding test set, published by OS-Copilot as separate desktop, mobile and web annotation files with a shared screenshot archive.

Release date not recorded · cross-platform
Benchmarks · curated

ScreenSpot-v2 instruction variants

Instruction variants over ScreenSpot-v2 screenshots that distinguish original commands, explicit actions, target descriptions and impossible operations.

Release date not recorded · cross-platform
Benchmarks · curated

SPA-Bench

340 single-app and cross-app tasks over 66 Chinese and English Android apps, with plug-in support for 11 smartphone agents and automatic coarse-to-fine success detection.

Release date not recorded · android
Benchmarks · curated

Spider2-V

Visual agent evaluation on data-science and engineering workflows.

Release date not recorded · web, desktop
Benchmarks · curated

ST-WebAgentBench

BrowserGym-based benchmark for evaluating the safety and policy compliance of enterprise web agents across six trustworthiness risk dimensions.

Release date not recorded · web
Benchmarks · curated

TheAgentCompany

Self-contained workplace environment for evaluating agents on professional tasks involving websites, files, code, and simulated coworkers.

Release date not recorded · web, desktop
Benchmarks · curated

VeriWeb (formerly VeriGUI)

Verifiable web information-seeking benchmark renamed from VeriGUI; desktop evaluation is described as future work.

Release date not recorded · web
Benchmarks · curated

VideoGUI

Benchmark for desktop GUI tasks derived from instructional videos.

Release date not recorded · desktop
Benchmarks · curated

VisualAgentBench

Evaluation framework spanning visual GUI, embodied, and design tasks, including mobile and WebArena-based environments.

Release date not recorded · cross-platform, web, android
Benchmarks · curated

VisualWebArena

Visually grounded web tasks extending the WebArena evaluation setting.

Release date not recorded · web
Benchmarks · curated

VisualWebBench

Evaluation of multimodal models on visual web understanding tasks.

Release date not recorded · web
Benchmarks · curated

WeaveBench

Computer-use benchmark mixing graphical interaction and code execution.

Release date not recorded · desktop, linux
Benchmarks · curated

Web Context Bench (Python)

Local-page benchmark comparing six browser representations on token use, factual reading and target-element selection, with deterministic answer scoring.

Release date not recorded · web
Benchmarks · curated

WebArena

Self-hosted web environments and tasks for evaluating autonomous web agents.

Release date not recorded · web
Benchmarks · curated

WebCanvas

Framework and evaluation resources for real-world web agents.

Release date not recorded · web
Benchmarks · curated

WebClick

A web-interface localization benchmark with screenshot instructions and target boxes, including agent browsing, human browsing and calendar cases.

Release date not recorded · web
Benchmarks · curated

WebGames

150 client-side web challenges in five capability domains, built to be easy for humans but hard for browsing agents, each revealing a password when solved.

Release date not recorded · web
Benchmarks · curated

WebGuard

Codebase, annotation toolkit, and action-level risk dataset for training and evaluating guardrail models that flag unsafe browser-agent actions before execution.

Release date not recorded · web
Benchmarks · curated

WebLINX

Demonstrations and evaluation for conversational web navigation.

Release date not recorded · web
Benchmarks · curated

WebRewardBench

An offline preference-evaluation set for web process reward models, pairing page and trajectory context with preferred and rejected candidate decisions.

Release date not recorded · web
Benchmarks · curated

WebShop

Simulated shopping environment for instruction-following web agents.

Release date not recorded · web
Benchmarks · curated

WebTailBench

Live-web task definitions and rubrics for evaluating computer-use agents, with preserved original tasks, a refreshed V2 task suite and a separate refusal set.

Release date not recorded · web
Benchmarks · curated

WebTestBench

An end-to-end web-testing benchmark where agents derive test checklists from development instructions, interact with deployed applications and report functional or constraint-related defects.

Release date not recorded · web
Benchmarks · curated

WebVoyager

Multimodal web navigation agent and evaluation tasks on real websites.

Release date not recorded · web
Benchmarks · curated

Windows Agent Arena

Windows 11 benchmark for multimodal desktop agents in a reproducible VM that runs locally or in parallel on Azure ML, with a harder mode where agents must set tasks up themselves.

Release date not recorded · windows, desktop
Benchmarks · curated

WindowsAgentArena-V2

Windows benchmark revision with task/evaluator fixes and VM-initialization support, documented alongside PC-Agent-E.

Release date not recorded · desktop, windows
Benchmarks · curated

WindowsWorld

Cross-application Windows benchmark with intermediate task evaluation.

Release date not recorded · desktop, windows
Benchmarks · curated

WONDERBREAD

An offline benchmark of workflow documentation, quality assessment and knowledge transfer using human web-navigation demonstrations, action traces and written procedures.

Release date not recorded · web
Benchmarks · curated

WorkArena

Browser-based enterprise workflow tasks built around ServiceNow.

Release date not recorded · web
Benchmarks · curated

WorldGUI

611 desktop GUI tasks that agents must complete from varied starting points, shipped with instructional videos and project files, plus the self-reflective WorldGUI-Agent baseline.

Release date not recorded · desktop

Search all records in this category · All categories