OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents
Extends OSWorld with tasks covering user misuse, prompt injection, and model misbehavior, with assessment of both task success and safety.
Desktop
Evaluation
Source-reviewed research and assets tagged with Evaluation.
Membership follows reviewed tags; it does not establish a dependency between neighboring records. Dates preserve source precision. Annotations retain their original language.
GUI-Robust: A Comprehensive Dataset for Testing GUI Agent Robustness in Real-World Anomalies
Builds GUI task data with real-world interaction anomalies and evaluates how agent behavior changes under those conditions.
Cross-platform
AgentSynth: Scalable Task Generation for Generalist Computer-Use Agents
AgentSynth composes easily generated subtasks into longer computer-use tasks and trajectories with adjustable difficulty. Its evaluation uses the number of subtasks to probe how agent success changes as task complexity grows.
DesktopWeb
Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling System
UI-NEXUS evaluates compositional mobile tasks in local utility apps and online Chinese and English service apps. AGENT-NEXUS decomposes those long tasks into self-contained atomic subtasks to study generalization beyond isolated screens or short action chains.
FingerTip 20K: A Benchmark for Proactive and Personalized Mobile LLM Agents
FingerTip 20K gathers 20K human Android demonstrations from users' long-term everyday app use, with user context, to evaluate proactive task suggestion and preference-aware task execution.
Android
EconWebArena: Benchmarking Autonomous Agents on Economic Tasks in Realistic Web Environments
Introduces EconWebArena, a curated benchmark of economic-data tasks requiring live website navigation, multimodal interpretation, and precise extraction. It evaluates visual grounding, planning, and interaction design within multistep workflows.
Web
GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior
GUI-Reflection synthesizes reflection and error-correction examples from successful trajectories and adds dedicated learning stages for these behaviors. It introduces a reflection task suite, a mobile environment for online data collection, and iterative reflection tuning.
MCPWorld: A Unified Benchmarking Testbed for API, GUI, and Hybrid Computer Use Agents
MCPWorld evaluates API, GUI, and hybrid computer-use agents with 201 tasks in applications whose source can be instrumented and extended with MCP support. It verifies completion from application behavior and packages the testbed in containers with GPU support.
Desktop
GUIPilot: A Consistency-based Mobile GUI Testing Approach for Detecting Application-specific Bugs
Introduces GUIPilot, which compares mobile design mock-ups with implemented screens through widget alignment and tests expected transitions by executing inferred UI actions. It evaluates both visual inconsistencies and workflow-specific bugs.
BIMgent: Towards Autonomous Building Modeling via Computer-use Agents
BIMgent plans software-specific workflows and executes GUI actions for building-information modeling. It evaluates multimodal building design and reconstruction tasks in professional authoring software.
macOSWorld: A Multilingual Interactive Benchmark for GUI Agents
macOSWorld provides 202 interactive tasks across 30 applications, with instructions and operating-system interfaces in five languages. It evaluates GUI agents on macOS workflows and includes a separate subset for deception-attack testing.
macOSDesktop
DPO Learning with LLMs-Judge Signal for Computer Use Agents
Uses an LLM judge to evaluate and filter synthetic GUI interaction trajectories for training a compact vision-language agent. OSWorld experiments assess the policy designed for local execution.
Desktop
Surfer-H Meets Holo1: Cost-Efficient Web Agent Powered by Open Weights
Describes Surfer-H, a web agent paired with the Holo1 vision-language model family trained on web content, synthetic examples, and agent-generated data. The work introduces WebClick for web-interface localization and evaluates browser task execution on WebVoyager.
Web
DeepShop: A Benchmark for Deep Research Shopping Agents
DeepShop evaluates shopping agents with evolving query complexity across product attributes, filters and sorting preferences. Its evaluation combines checks of these individual requirements with overall task success.
Web
VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agents
Introduces VPI-Bench, interactive web-platform variants containing visually embedded malicious instructions. It evaluates computer-use and browser-use agents under these threats and examines system-prompt defenses.
Web
MLA-Trust: Benchmarking Trustworthiness of Multimodal LLM Agents in GUI Environments
Benchmarks truthfulness, controllability, safety, and privacy of multimodal LLM agents across interactive web and mobile GUI tasks.
WebAndroidCross-platform
RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agents
Introduces RiOSWorld, a computer-use risk benchmark separating user-originated and environmental threats. It evaluates both risky-goal intent and goal completion across web and other computer applications.
Web
Open CaptchaWorld: A Comprehensive Web-based Platform for Testing and Benchmarking Multimodal LLM Agents
Open CaptchaWorld evaluates multimodal agents on interactive browser-based CAPTCHA puzzles. Its benchmark distinguishes task success from the depth of visual reasoning and interaction needed to complete a puzzle.
Web
RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environments
RedTeamCUA combines a virtual-machine OS environment with containerized websites for controlled prompt-injection tests spanning both interfaces. Its RTC-Bench contains 864 examples and can initialize agents at injection points to separate attack susceptibility from navigation failures.
WebDesktop
AdInject: Real-World Black-Box Attacks on Web Agents via Advertising Delivery
Studies advertising delivery as an indirect-injection channel for web agents under a black-box threat model. AdInject evaluates how static ad content shaped by website context can redirect browser actions.
Web
XBOUND: Exploring Capability Boundaries of Device-Control Agents at the State Level
XBOUND evaluates GUI agents at the level of a screen state, considering multiple instructions and possible target actions within it. The study separates action matching from instruction unification and compares effects of grounding and trajectory data.
ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows
ScienceBoard combines an interactive environment with professional scientific software and 169 human-curated tasks in areas including biochemistry, astronomy, and geoinformatics. It evaluates computer-use agents on visually rich research workflows and analyzes their execution limitations.
Desktop
Web-Shepherd: Advancing PRMs for Reinforcing Web Agents
Web-Shepherd is a process reward model for assessing individual steps of web-navigation trajectories. It introduces a collection of 40,000 step-level preference pairs with checklists and a separate benchmark for comparing reward models, then evaluates verification during WebArena-lite navigation.
Web
EVA: Evolving Semantic Adversaries for Red-Teaming GUI Agents Against Environmental Injection Attacks
Introduces EVA, a red-teaming framework that evolves semantic payloads for GUI environmental-injection tests. It studies the contribution of semantic versus visual deception and distills discovered patterns into reusable evaluation rules.
Efficient Agent Training for Computer Use
PC Agent-E augments 312 human computer-use trajectories with alternative action decisions synthesized by Claude 3.7 Sonnet. It studies training with this combination and introduces WindowsAgentArena-V2 as an updated evaluation benchmark.
WindowsDesktop
Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis
Introduces OSWorld-G grounding evaluation and the Jedi synthesis pipeline for examples spanning layout reasoning and fine-grained interaction.
Desktop
The Hidden Dangers of Browsing AI Agents
Outlines an end-to-end threat model and empirical security evaluation across architectural layers of autonomous browser agents.
Web
From Assistants to Adversaries: Exploring the Security Risks of Mobile LLM Agents
Analyzes security vulnerabilities across OEM system-level, third-party, and research mobile LLM agents, covering GUI and cross-application attack surfaces.
GEM: Gaussian Embedding Modeling for Out-of-Distribution Detection in GUI Agents
GEM detects instructions outside a GUI agent’s capabilities by modeling distances in its input-embedding space with a Gaussian mixture. Experiments across mobile, computer and browser data also examine selective escalation to a cloud model when an input is detected as unfamiliar.
DesktopWebCross-platform
Mobile-Bench-v2: A More Realistic and Comprehensive Benchmark for VLM-based Mobile Agents
Mobile-Bench-v2 evaluates mobile GUI agents using multiple acceptable offline action paths, noisy interfaces with ads or pop-ups, and ambiguous instructions with scripted clarification exchanges. These splits separate step-level execution, robustness, and proactive interaction.
Characterizing Unintended Consequences in Human-GUI Agent Collaboration for Web Browsing
Studies unintended consequences of collaborative web browsing with GUI agents through social-media analysis and interviews. It organizes observed failures, user impacts, and mitigation practices around planning, interface interaction, feedback, privacy, and trust.
Web
Do MLLMs Capture How Interfaces Guide User Behavior? A Benchmark for Multimodal UI/UX Design Understanding
Introduces WiserUI-Bench, pairing real UI variants from industry A/B tests with observed behavioral winners and expert explanations. It evaluates models’ ability to predict the more effective interface and explain its effect on user behavior.
Adjacent research
OSUniverse: Benchmark for Multimodal GUI-navigation AI Agents
Introduces OSUniverse, a desktop GUI benchmark spanning precise clicks through multistep and multiapplication tasks. It provides both manual scoring and an automated validation mechanism.
Desktop
Toward a Human-Centered Evaluation Framework for Trustworthy LLM-Powered GUI Agents
Presents a position paper on privacy and security evaluation for GUI agents, identifying risk categories and challenges in involving human evaluators. It proposes risk-aware assessment and in-context consent as design directions.
WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks
Introduces WASP, an end-to-end benchmark for prompt-injection security in web agents. Its evaluation separates partial attack progress from completion of the attacker’s full objective.
Web
DoomArena: A framework for Testing AI Agents Against Evolving Security Threats
Presents a modular security testing framework that plugs into BrowserGym and tool-use benchmarks to evaluate agents under configurable threat models.
Web
LearnAct: Few-Shot Mobile GUI Agent with a Unified Demonstration Benchmark
LearnAct parses human demonstrations into reusable knowledge, retrieves relevant examples, and uses them during mobile GUI task execution. The accompanying LearnGUI benchmark includes 2,252 offline tasks and 101 online tasks for studying adaptation from demonstrations.
VLM-Fuzz: Vision Language Model Assisted Recursive Depth-first Search Exploration for Effective UI Testing of Android Apps
VLM-Fuzz combines depth-first UI exploration with Android manifest analysis, runtime widget hierarchies, and on-demand visual reasoning to generate app test inputs. It evaluates coverage on 59 benchmark apps and uses additional Google Play apps to study crash discovery.
Android
The Obvious Invisible Threat: LLM-Powered GUI Agents' Vulnerability to Fine-Print Injections
Characterizes six forms of GUI-content injection and studies their effects on GUI agents and human users using adversarial webpages. The analysis examines privacy failures and the limits of human oversight.
Web
UI-E2I-Synth: Advancing GUI Grounding with Large-Scale Instruction Synthesis
UI-E2I-Synth uses GPT-4o to synthesize GUI grounding instructions with varied complexity, addressing small elements, imbalanced element types, and implicit requests. The accompanying UI-I2E-Bench evaluates instruction-to-element localization across these annotation challenges.
RealWebAssist: A Benchmark for Long-Horizon Web Assistance with Real-World Users
RealWebAssist evaluates web assistance through sequences of instructions collected from real users across multiple websites. It tests intent interpretation, user-specific routines, long-term context, and visual grounding when instructions are ambiguous or evolve during interaction.
Web
Breaking the Data Barrier -- Building GUI Agents Through Task Generalization
Studies whether mid-training on GUI perception, multimodal reasoning, and text reasoning transfers to GUI-agent planning. Experiments compare 11 task sources and curated mixtures using WebArena and AndroidWorld to investigate training-data choices when GUI trajectories are scarce.
WebAndroid
AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories
AgentRewardBench evaluates automated web-agent judges against expert reviews of 1,302 trajectories from five benchmarks. The annotations cover completion, side effects, and repetition, enabling comparison of LLM judges and rule-based task evaluators.
Web
On the Robustness of GUI Grounding Models Against Image Attacks
Evaluates GUI grounding models under natural image noise, untargeted adversarial perturbations, and targeted image attacks. Experiments cover mobile, desktop, and web interfaces and analyze sensitivity to image resolution and adversarial changes.
WebDesktopCross-platform
ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use
Evaluates GUI target localization in high-resolution professional software and studies cascaded visual search to narrow the target region.
Desktop
ReuseDroid: A VLM-empowered Android UI Test Migrator Boosted by Active Feedback
ReuseDroid migrates Android GUI tests between functionally similar apps using VLM agents that separate shared test intent from app-specific interaction flows. It introduces LinPro, a dataset of 578 migration tasks across 39 apps, to evaluate test transfer.
Android
Are Autonomous Web Agents Good Testers?
Adapts web agents to execute natural-language test steps, verify assertions, and issue test verdicts. The study introduces 113 passing and failing test cases across three offline web applications and compares two agent implementations with qualitative failure analysis.
Web
An Illusion of Progress? Assessing the Current State of Web Agents
Introduces Online-Mind2Web, with 300 tasks across 136 live websites, and examines how benchmark design affects assessments of web agents. It also develops an LLM judge for task completion and compares its judgments with human evaluation.
Web
Navi-plus: Managing Ambiguous GUI Navigation Tasks with Follow-up Questions
Navi-plus evaluates GUI agents that ask follow-up questions to complete ambiguous user instructions. It provides GUI clarification question-answer pairs and Dual-Stream Trajectory Evaluation for this interactive setting.
sudo rm -rf agentic_security
Evaluates security failures of computer-use agents under adversarial requests and screen-context manipulation. The study examines refusal behavior while agents perform tasks in desktop and web environments.
DesktopWeb
Agent-Initiated Interaction in Phone UI Automation
Defines when phone agents should request user interaction and introduces AndroidInteraction annotations for deciding whether and how to engage the user.
Android
Does Chain-of-Thought Reasoning Help Mobile GUI Agent? An Empirical Study
This study compares matched reasoning and non-reasoning vision-language models on static mobile GUI tasks and interactive AndroidWorld navigation. It finds that chain-of-thought effects vary across tasks and model families.
Android
UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction
Provides annotated desktop demonstrations for evaluating element grounding, layout grounding, and action prediction.
Desktop
STEVE: A Step Verification Pipeline for Computer-use Agent Training
Labels individual computer-use actions by comparing surrounding screenshots, then trains agents from both successful and unsuccessful steps.
DesktopWindows
AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agents
Evaluates whether autonomous web agents minimize use of sensitive personal information when completing tasks that expose unnecessary private data.
Web
In-Context Defense in Computer Agents: An Empirical Study
Studies in-context defenses for computer agents exposed to misleading pop-ups and other environmental instructions. It examines defensive reasoning before action planning and how a small set of examples changes resistance to these attacks.
BEARCUBS: A benchmark for computer-using web agents
BEARCUBS provides 111 live-web information-seeking questions with short answers and human-validated browsing trajectories. Its tasks require multimodal computer interaction, including video understanding and navigation of interactive 3D content.
Web
FedMABench: Benchmarking Mobile Agents on Decentralized Heterogeneous User Data
Introduces a benchmark for federated mobile-agent training under heterogeneous user and application distributions, combining datasets, algorithms, and base models.
SafeArena: Evaluating the Safety of Autonomous Web Agents
Pairs benign and harmful web tasks to assess deliberate misuse risks and categorize agent behavior by risk level.
Web
AutoEval: A Practical Framework for Autonomous Evaluation of Mobile Agents
Generates reward signals from UI-state changes and uses a judging system to evaluate mobile agents with reduced manual evaluator construction.
Showing 421–480 of 720 resources.
Get full records and evidence through the topic export manifest. Source records can be passed to the standard RAG and citation exporters after cloning the catalog.