arXiv v1
VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents
Benchmarks visual agents across GUI, embodied, and design tasks, with training trajectories collected through solvers, agents, and human demonstrations.
RESEARCH LIBRARY
Computer-use research library
Source-backed papers on GUI grounding, agents, training, evaluation, and safety.
Read the research survey ↗ · Methods, evaluation, and human agency — an English working manuscript with a source-by-source reading register.
PAPER TOPICSEvaluationData generationGUI groundingWeb navigationPlanningMobile navigationReinforcement learningSafety & securityAll topics ↗
AI-assisted editorial review of primary sources; experiments are not independently reproduced. Dates keep their recorded precision. Resource links below follow reviewed introduction or documentation relationships. Method ↗
arXiv v4
AppAgent v2: Advanced Agent for Flexible Mobile Interactions
Separates exploration and deployment while using a structured application knowledge base and retrieval to guide mobile workflows.
arXiv v3
Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractions
Measures how irrelevant GUI content affects multimodal agents using a simulated dataset and three perception patterns. The study also examines adversarial environment injection and possible ways to improve instruction faithfulness.
arXiv v1
OmniParser for Pure Vision Based GUI Agent
Parses screenshots into interactable regions and semantic element descriptions to support visually grounded agent actions.
arXiv v1
OfficeBench: Benchmarking Language Agents across Multiple Applications for Office Automation
Evaluates language agents on office workflows requiring information integration, long-term planning, and switching between applications.
arXiv v1
AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents
Introduces simulated application APIs and state-based task evaluation for interactive coding agents performing everyday digital workflows.
arXiv v2
AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?
Introduces realistic open-web tasks and the SeePlanAct agent, analyzing the limitations of browsing and retrieval in time-consuming assistant work.
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing ↗
arXiv v3
IDA: Breaking Barriers in No-code UI Automation Through Large Language Models and Human-Centric Design
IDA helps business users create web UI automation through guided demonstrations, a semantic programming model, and a teaching-oriented interface. A prototype user study examines automation creation with enterprise applications by users without programming backgrounds.
arXiv v1
Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systems
Agent-E combines hierarchical web-task control with DOM distillation and observations of interface changes. WebVoyager experiments inform a broader discussion of primitive skills, observation reduction, and self-improvement in agent architecture.
arXiv v1
Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?
Benchmarks multimodal agents on data-science and engineering workflows involving enterprise applications, code, and GUI operations in real computer environments.
arXiv v3
Systematic Categorization, Construction and Evaluation of New Attacks against Multi-modal Mobile GUI Agents
Develops a threat taxonomy and evaluation framework for attacks against multimodal mobile GUI agents, with case studies and dataset experiments.
arXiv v1
AUITestAgent: Automatic Requirements Oriented GUI Function Testing
Automates mobile GUI tests from natural-language requirements, combining action generation with verification based on interaction traces.
arXiv v3
UICrit: Enhancing Automated Design Evaluation with a UICritique Dataset
Introduces designer-authored mobile UI critiques and quality ratings, studying how targeted examples improve automated design feedback.
arXiv v2
WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks
Defines compositional enterprise workflows to evaluate planning and reasoning, with generation of ground-truth interaction traces.
arXiv v3
MobileFlow: A Multimodal LLM For Mobile GUI Agent
Introduces a mobile GUI model using hybrid visual encoders, mixture-of-experts expansion, and alignment training for multilingual interfaces.
arXiv v1
MobileExperts: A Dynamic Tool-Enabled Agent Team in Mobile Devices
Introduces MobileExperts, which assembles mobile-agent teams, lets agents explore and formulate tools, and coordinates them through two levels of planning. A benchmark organizes tasks by hierarchical intelligence levels.
arXiv v2
AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents
Introduces Android screenshots with element locations, functionality descriptions, and instructions linked to sequences of GUI actions.
Findings of the Association for Computational Linguistics: ACL 2025 ↗
arXiv v2
Seeing is Believing: Vision-driven Non-crash Functional Bug Detection for Mobile Apps
Trident combines exploration, monitoring, and detection agents to identify functional bugs that do not crash a mobile app. It reasons over screenshot sequences and interface transitions to construct test oracles, evaluated on 590 non-crash bugs and additional live apps.
arXiv v4
CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents
Introduces CRAB, a framework for cross-environment agent tasks with graph-based evaluation and a benchmark spanning desktop and phone interactions.
Findings of the Association for Computational Linguistics: ACL 2025 ↗
arXiv v4
Tree Search for Language Model Agents
Applies best-first search to agent actions within interactive web environments, allowing exploration and multistep planning at inference time. WebArena and VisualWebArena experiments examine search behavior and test-time computation.
arXiv v1
Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents
Evaluates mobile agents using combined UI and API actions, tasks with varying app scope, and checkpoints for intermediate progress.
arXiv v2
Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding
Introduces Screen Point-and-Read and a hierarchical layout approach that interprets screen regions indicated by a user-selected point.
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing ↗
arXiv v3
Identifying User Goals from UI Trajectories
Defines goal identification from observed UI trajectories and an evaluation method for comparing intent descriptions within the same interface context. Experiments reuse Android and web automation data to compare human and model inference of users’ detailed goals.
arXiv v3
E-ANT: A Large-Scale Dataset for Efficient Automatic GUI NavigaTion
Introduces Chinese-language GUI navigation traces and annotated screenshots collected from human interactions with mini-applications.
arXiv v3
VGA: Vision GUI Assistant -- Minimizing Hallucinations through Image-Centric Fine-Tuning
Uses image-dependent question-answer data and staged fine-tuning to improve GUI comprehension and reduce unsupported visual answers.
Findings of the Association for Computational Linguistics: EMNLP 2024 ↗
arXiv v1
GUI Action Narrator: Where and When Did That Action Take Place?
Introduces Act2Cap and a cursor-guided method for describing GUI actions from video, with temporal and spatial localization challenges.
arXiv v2
WONDERBREAD: A Benchmark for Evaluating Multimodal Foundation Models on Business Process Management Tasks
Introduces WONDERBREAD, a benchmark pairing human WebArena workflow demonstrations with screen recordings, action logs, and written procedures. Six evaluation tasks cover documentation, knowledge transfer, and process improvement beyond end-to-end execution.
arXiv v3
Dissecting Adversarial Robustness of Multimodal LM Agents
Introduces adversarial tasks on VisualWebArena and an evaluation framework that tracks how manipulated information passes between agent components. The analysis examines vulnerabilities introduced by reflection and tree-search evaluators as well as direct action generation.
arXiv v3
WebCanvas: Benchmarking Web Agents in Online Environments
Introduces WebCanvas and Mind2Web-Live, evaluating online task progress through meaningful intermediate states on changing websites.
arXiv v2
GUICourse: From General Vision Language Models to Versatile GUI Agents
Introduces GUIEnv, GUIAct, and GUIChat datasets to develop OCR, grounding, component knowledge, and interaction abilities in visual-language models.
arXiv v2
GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding
Introduces video-based GUI questions and annotations for studying temporal interface understanding across varied interaction settings.
arXiv v1
DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning
Combines offline initialization and online reinforcement learning in Android environments, with automated evaluation and curriculum-based sampling.
arXiv v1
VideoGUI: A Benchmark for GUI Automation from Instructional Videos
Evaluates GUI automation from instructional videos at separate levels of high-level planning, action narration, and primitive execution.
arXiv v3
GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning
Introduces GuardAgent, which translates safety requests into executable guardrail code using plans and retrieved demonstrations. Mind2Web-SC supplies a web-agent safety-policy evaluation alongside a separate healthcare access-control benchmark.
arXiv v2
GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices
Introduces annotated cross-application mobile trajectories and studies history-aware navigation over long task sequences.
arXiv v1
Practical, Automated Scenario-based Mobile App Testing
ScenTest builds scenario-specific event knowledge graphs from crowdsourced test reports containing GUI and action information. It uses the current app state and testing context to generate scenario tests across applications.
arXiv v1
MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents
Introduces a mobile-agent benchmark with tasks across open-source applications, organized by difficulty to reduce manual evaluation effort.
arXiv v2
Tell Me What's Next: Textual Foresight for Generic UI Representations
Textual Foresight learns UI representations by describing the future screen expected after an action on the current interface. The OpenApp dataset supports pretraining and downstream evaluation of representations that connect individual elements with whole-screen changes.
arXiv v3
CAAP: Context-Aware Action Planning Prompting to Solve Computer Tasks with Front-End UI Only
CAAP converts screenshots into textual observations and plans keyboard-and-mouse actions using a prompt that examines task context from multiple perspectives. The agent is evaluated on MiniWoB++ and WebShop using only front-end visual observations.
arXiv v6
On the Effects of Data Scale on UI Control Agents
Studies how Android task performance changes with demonstration scale and releases AndroidControl with high- and low-level instructions.
arXiv v1
Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration
Separates mobile planning, decision-making, and reflection, with a memory unit for task progress and relevant content.
arXiv v1
WebSuite: Systematically Evaluating Why Web Agents Fail
WebSuite organizes browser operations into an action taxonomy and tests both isolated controls and complete workflows. Its diagnostic tasks attribute failures to specific actions, enabling comparison of text-based and multimodal web agents.
arXiv v2
Large Language Models Can Self-Improve At Web Agent Tasks
Examines self-improvement by fine-tuning web agents on three mixtures of model-generated training data. WebArena experiments and trajectory-level evaluation measures study task performance, robustness, and behavior beyond aggregate success rates.
arXiv v2
V-Zen: Efficient GUI Understanding and Precise Grounding With A Novel Multimodal LLM
Describes V-Zen, a multimodal GUI model with dual-resolution image encoders for grounding and next-action prediction, fine-tuned using GUIDE interface elements and task sequences.
arXiv v5
AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents
Introduces parameterized Android tasks with initialization, success checks, and cleanup for reproducible interactive evaluation.
arXiv v1
SmartFlow: Robotic Process Automation using LLMs
Presents SmartFlow, which converts visible GUI elements into text for language-model action planning and executes the plan through a scripting engine. A collection of enterprise applications evaluates adaptation to different layouts and tasks.
arXiv v1
Latent State Estimation Helps UI Agents to Reason
Studies explicit textual estimates of latent UI state, including whether commanded actions occurred and how far a task has progressed. The experiments compare agents that reason with these estimates against agents that omit them across public and internal task benchmarks.
arXiv v1
MUD: Towards a Large-Scale and Noise-Filtered UI Dataset for Modern Style UI Modeling
MUD combines LLM-guided Android app exploration, noise filtering and human annotation to collect modern UI screenshots. The study evaluates the resulting data for element detection and UI retrieval.
arXiv v2
Unveiling Disparities in Web Task Handling Between Human and Web Agent
Compares human think-aloud web-task behavior with web-agent planning, action, and reflection. The analysis highlights differences in updating plans, handling ambiguity, and investigating failures.
arXiv v3
Visual Grounding Methods for Efficient Interaction with Desktop Graphical User Interfaces
Compares modular OCR-based and end-to-end methods for locating desktop GUI targets from natural-language instructions.
arXiv v1
Navigating WebAI: Training Agents to Complete Web Tasks with Large Language Models and Reinforcement Learning
Combines supervised and reinforcement learning for MiniWoB web tasks and investigates whether agents understand HTML structure or memorize target elements.
arXiv v3
MMAC-Copilot: Multi-modal Agent Collaboration Operating Copilot
Introduces MMAC-Copilot, which coordinates agents with different expertise to operate PC applications. VIBench evaluates visual interaction with applications that lack usable APIs, spanning office, recreation and game tasks.
arXiv v3
Benchmarking Mobile Device Control Agents across Diverse Configurations
Introduces B-MoCA, an interactive mobile-control benchmark that varies device settings, layouts, and languages to test generalization.
arXiv v2
Grounded Language Agent for Product Search via Intelligent Web Interactions
Introduces GLAINTEL, a Flan-T5-based product-search web agent studied under unsupervised learning, demonstration-based training, and unsupervised domain adaptation. It compares ways to combine demonstrations with reinforcement learning.
arXiv v2
MMInA: Benchmarking Multihop Multimodal Internet Agents
Evaluates multimodal agents on compositional tasks across changing websites and studies replaying prior trajectories as memory support.
Findings of the Association for Computational Linguistics: ACL 2025 ↗
arXiv v4
Large Language Models for Mobile GUI Text Input Generation: An Empirical Study
Studies language-model generation of mobile text inputs, comparing UI representations, execution feedback, and human assistance for app testing.
arXiv v3
Enhancing Mobile "How-to" Queries with Automated Search Results Verification and Reranking
Executes instructions from mobile how-to search results inside a controlled Android environment, then uses the observed success of those instructions to verify and rerank technical-support results.
arXiv v2
LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Task Automation
LlamaTouch executes mobile tasks on devices and evaluates whether an agent visits manually annotated essential application or system states. Screenshot and hierarchy annotations, combined with exact and fuzzy state matching, support evaluation of 496 tasks across multiple agents.
arXiv v1
Training a Vision Language Model as Smartphone Assistant
Trains a smartphone assistant from sequences of screenshots paired with earlier actions. The model predicts UI-only interactions such as taps and swipes, with evaluation on Android in the Wild.
arXiv v2
OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
Introduces a real-computer evaluation environment with execution-based checks for tasks spanning desktop applications and operating-system state.
Showing 1,021–1,080 of 1,200, newest first.