Evaluation
Source-reviewed research and assets tagged with Evaluation.
Membership follows reviewed tags; it does not establish a dependency between neighboring records. Dates preserve source precision. Annotations retain their original language.
BrowserART
Red-teaming benchmark and test suite from Scale AI evaluating whether refusal-trained language models can be jailbroken when deployed as browser agents.
ColorBench
A graph-based mobile GUI benchmark that evaluates long tasks through screenshot states and multiple valid or erroneous transitions, including reflection, replanning and backtracking.
CUAVerifierBench
Human-annotated browser-agent trajectories for comparing automated task-success and process-quality verifiers against reviewer judgments.
EIA
Evaluation suite for Environmental Injection Attacks that embed invisible malicious HTML elements into benign websites to test privacy leakage in generalist web agents.
FedMABench
A federated mobile-agent training and evaluation suite that varies user-data heterogeneity and application distribution, with released partition annotations and inference/scoring scripts.
GUI-Robust
Evaluation scripts and linked data for comparing GUI action predictions and task execution under ordinary and anomalous interface conditions.
GUIHalluBench
Measures hallucination in UI-element parsing by matching the boxes and descriptions a model lists for 1,000 English and 1,000 Chinese screenshots against human annotations.
iOSWorld
Native iPhone-simulator benchmark with shared user data, application workflows, agent runners, and rubric-based scoring.
MacAgentBench
676 macOS tasks across 25 applications, each run in its own Docker container and scored by deterministic rule-based checks with multiple checkpoints.
Mind2Web 2
Long-horizon agentic web search benchmark whose Agent-as-a-Judge framework scores time-varying, citation-backed answers, with evaluation scripts for its dev and test sets.
MiniWoB++
Over 100 web-interaction environments extending OpenAI's MiniWoB, controlled from Python through a Gymnasium API that drives the browser with Selenium.
MLA-Trust
Unified evaluation framework measuring truthfulness, controllability, safety, and privacy of multimodal LLM agents across interactive web and mobile GUI tasks.
MMBench-GUI
Four-level GUI-agent benchmark across six platforms, from content understanding and element grounding to task automation and collaboration, scored with an efficiency-quality area metric.
MMInA
Multimodal web tasks in a WebArena-based environment that chain from 1 to 10 hops across sites such as a shopping store and Wikipedia, testing long-chain reasoning.
MobBench task definitions
A public set of real-device mobile task definitions used to evaluate MobA, with Chinese and English task material packaged in a versioned JSON release.
Mobile-Bench-v2 released data
Offline mobile-agent task data spanning common tasks and released noisy, contaminated and ambiguous-instruction subsets.
Mobile-Eval-E task definitions
Task definitions for Mobile-Agent-E’s long-horizon, multi-app Android evaluation, packaged as a 25-record test split with rubrics and human-reference operation strings.
MobileAgentBench
Automated benchmark for mobile LLM agents whose default tasks use ten Simple Mobile Tools Android apps, installed as a Python library that detects task completion.
MobileGUI-VBench
Mobile-app task definitions covering everyday scenarios, with annotated preconditions and clarification requirements for evaluating agent behavior.
MobileGym
A browser-hosted mobile-app simulator with programmable state, parameterized tasks and deterministic state-checking judges for parallel GUI-agent evaluation and online reinforcement learning.
MobileGym++ / MobileGym++ Bench
Browser-rendered mobile simulation with resettable app state and a shared task suite for GUI-only and GUI-plus-tool agents.
MobileSafetyBench
Android emulator testbed that scores mobile device-control agents on both safety and helpfulness, using ADB and Appium for observations, actions and success checks.
MobileWorld (Tongyi-MAI)
A containerized Android evaluation environment with GUI-only, agent-user interaction and MCP-augmented tasks, plus checks on application and backend state.
MobileWorldBench
Benchmark of semantic world modeling for mobile agents, with question-answering tasks and GPT-judged generation tasks, plus training data built on AitW and AndroidControl screens.
Online Mind2Web
300 tasks on 136 live websites for evaluating web agents online, with outdated tasks replaced over time, human-reviewed submissions and the WebJudge automatic evaluator.
OpenApps
Configurable browser apps and task rewards for testing computer-use agents under changes in content, layout and appearance.
OS-Critic Bench
A cross-platform GUI-critic benchmark with mobile, desktop and web JSONL splits and screenshot assets, accompanied by public inference and accuracy-scoring scripts.
OS-Harm
Benchmark for safety of computer-use agents across misuse, prompt injection, and unintended model behavior.
OS-Map
An OSWorld-derived desktop benchmark with an Ubuntu virtual-machine environment and expanded task configurations for measuring breadth and depth of computer interaction.
OSWorld
Execution-based evaluation of agents performing tasks in real desktop environments.
OSWorld-G
Computer-use grounding benchmark with original and refined instructions, released with the Jedi synthetic data pipeline, a 4-million-example Jedi dataset and Jedi-3B and Jedi-7B models.
OSWorld-Human
Human reference trajectories and scoring code for measuring the action efficiency of computer-use agents on OSWorld tasks.
PPTC
PowerPoint creation and editing benchmark with API-based execution and checks of the resulting presentation files.
Precision CUA Bench
Evaluation resources for pixel-precise cursor grounding in code editors.
RecreationBench
Tasks for recreating reference applications with hybrid computer interaction.
RiOSWorld
Measures risky behavior of multimodal computer-use agents in an Ubuntu virtual machine, with risk scenarios such as phishing emails and released evaluation trajectories.
SaaS-Bench
106 browser-driven business workflows across 23 self-hosted SaaS apps in six domains, scored by per-task verifiers that read each app's database and give partial credit.
ScienceBoard
Scientific-workflow tasks for multimodal agents in desktop software such as ChimeraX, KAlgebra, Celestia, Grass GIS and Lean 4, with apps modified to expose state for evaluation.
Screen Point-and-Read / Tree-of-Lens
Screen-reading benchmark and Tree-of-Lens implementation for interpreting user-indicated regions and their layout context.
ScreenSpot
GUI grounding test set of 1,272 screenshot instructions from iOS, Android, macOS, Windows and web interfaces, each labeled with the target element's bounding box.
ScreenSpot Pro
GUI grounding benchmark focused on professional high-resolution interfaces.
ScreenSpot v2
Second version of the ScreenSpot grounding test set, published by OS-Copilot as separate desktop, mobile and web annotation files with a shared screenshot archive.
ScreenSpot-v2 instruction variants
Instruction variants over ScreenSpot-v2 screenshots that distinguish original commands, explicit actions, target descriptions and impossible operations.
SPA-Bench
340 single-app and cross-app tasks over 66 Chinese and English Android apps, with plug-in support for 11 smartphone agents and automatic coarse-to-fine success detection.
ST-WebAgentBench
BrowserGym-based benchmark for evaluating the safety and policy compliance of enterprise web agents across six trustworthiness risk dimensions.
TheAgentCompany
Self-contained workplace environment for evaluating agents on professional tasks involving websites, files, code, and simulated coworkers.
VeriWeb (formerly VeriGUI)
Verifiable web information-seeking benchmark renamed from VeriGUI; desktop evaluation is described as future work.
VisualAgentBench
Evaluation framework spanning visual GUI, embodied, and design tasks, including mobile and WebArena-based environments.
Showing 601–660 of 720 resources.
Get full records and evidence through the topic export manifest. Source records can be passed to the standard RAG and citation exporters after cloning the catalog.