arXiv v3
OpenCUA: Open Foundations for Computer-Use Agents
Presents demonstration collection infrastructure, the AgentNet dataset, and a training pipeline for open computer-use models.
RESEARCH LIBRARY
Computer-Use 论文库
从 GUI 定位到智能体训练、评测与安全，每篇都可追溯到原始论文。
阅读英文研究综述 ↗ · 方法、评测与人的控制权；英文工作稿附有逐篇阅读记录。
条目经过 AI 辅助的一手来源编辑审阅，实验未独立复现。发表日期保留来源记录的精度。下方资源链接来自已审阅的介绍或文档关系。 收录方法 ↗
arXiv v2
MVISU-Bench: Benchmarking Mobile Agents for Real-World Tasks by Multi-App, Vague, Interactive, Single-App and Unethical Instructions
MVISU-Bench evaluates bilingual mobile tasks involving cross-app work, vague requests, user interaction and unethical instructions alongside ordinary single-app tasks. Its Aider prompting module is designed to clarify intent and mitigate risky execution within agent workflows.
arXiv v2
InfiGUI-G1: Advancing GUI Grounding with Adaptive Exploration Policy Optimization
Uses multiple candidate answers and adaptive exploration rewards to improve spatial and semantic alignment in GUI grounding.
arXiv v2
Test-Time Reinforcement Learning for GUI Grounding via Region Consistency
GUI-RC aggregates multiple grounding predictions into spatial voting grids to choose a consensus region. GUI-RCPO then turns agreement with that region into a reward for test-time policy updates on unlabeled GUI screenshots.
arXiv v1
Cognitive Duality for Adaptive Web Agents
Introduces CogniWeb, which switches between fast reactive processing and slower deliberative reasoning according to task complexity. WebArena experiments examine this combination of learned behaviors and online planning.
arXiv v2
SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience
SEAgent explores unfamiliar software using a world-state assessor and an automatically generated task curriculum. It learns from failed and successful interactions through adversarial imitation and GRPO, then combines specialist experience into a generalist agent evaluated on five OSWorld software environments.
arXiv v1综述
OS Agents: A Survey on MLLM-based Agents for General Computing Devices Use
Surveys agents that operate computers and phones through operating-system interfaces, covering components, capabilities, foundation models, frameworks, and benchmarks.
跨平台综述
arXiv v4
Beyond Pixels: Exploring DOM Downsampling for LLM-Based Web Agents
D2Snap reduces serialized DOM observations while preserving actionable elements and features that distinguish their uses. The study compares reduced DOM inputs with screenshot-based observations for a web agent on tasks sampled from Online-Mind2Web.
arXiv v1
GuirlVG: Incentivize GUI Visual Grounding via Empirical Exploration on Reinforcement Learning
Introduces GuirlVG, an empirical study of reinforcement fine-tuning for GUI visual grounding. It analyzes training components and adds an adaptive KL-based mechanism to limit reward over-optimization.
arXiv v2
Evolving in Tasks: Empowering the Multi-modality Large Language Model as the Computer Use Agent
The Self-Evolution Agent combines automatic generation of verifiable computer-use trajectories with step-wise reinforcement learning for long-horizon training. It also describes a model-enhancement procedure that integrates grounding and planning capabilities without an additional training stage.
arXiv v1
Uncertainty-Aware GUI Agent: Adaptive Perception through Component Recommendation and Human-in-the-Loop Refinement
RecAgent narrows attention to recommended UI components and requests user feedback when task intent or decisions remain uncertain. It introduces ComplexAction to evaluate specified single-step GUI actions in difficult interface scenarios.
arXiv v3
CoAct-1: Computer-using Multi-Agent System with Coding Actions
CoAct-1 delegates subtasks between a GUI operator and a programmer that executes Python or Bash. Its OSWorld evaluation studies a hybrid action space in which scripts handle suitable file and data operations while visual interaction handles other steps.
arXiv v4
NaviMaster: Learning a Unified Policy for GUI and Embodied Navigation Tasks
NaviMaster trains a shared policy for GUI and embodied navigation using a common visual-trajectory formulation. It combines trajectory collection, reinforcement learning and distance-aware rewards, with evaluation across both task families.
arXiv v3
Web-CogReasoner: Towards Multimodal Knowledge-Induced Cognitive Reasoning for Web Agents
Web-CogReasoner separates factual and conceptual web knowledge from procedural reasoning, using a structured curriculum to train a multimodal agent. Web-CogDataset and Web-CogBench support study of these capabilities alongside web-navigation evaluation.
arXiv v4
NaturalGAIA: A Verifiable Benchmark and Hierarchical Framework for Long-Horizon GUI Tasks
NaturalGAIA builds verifiable long-horizon GUI tasks from human interaction intents while separating causal task structure from its verbal description. Its LightManus-Jarvis framework divides dynamic planning and context management from execution based on visual and structural interface information.
arXiv v1
Phi-Ground Tech Report: Advancing Perception in GUI Grounding
Reports a GUI-grounding model family and empirical design choices covering source data, perception training, and coordinate prediction.
arXiv v1
Magentic-UI: Towards Human-in-the-loop Agentic Systems
Magentic-UI combines browser interaction, code execution, file operations, and MCP tools in an interface for human-agent collaboration. It studies collaborative planning and task execution, action guards, and persistent memory through benchmark tasks, simulated users, user studies, and safety assessments.
arXiv v4
UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding
Combines grounding-oriented reward and sampling improvements with inference-time image decomposition and target selection.
arXiv v1
MapAgent: Trajectory-Constructed Memory-Augmented Planning for Mobile Task Automation
Builds reusable page memories from past mobile trajectories and retrieves them to support task planning and execution.
arXiv v1
MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents
MMBench-GUI evaluates interface understanding, element grounding, task automation, and collaboration across six named desktop, mobile, and web platforms. It introduces the Efficiency-Quality Area metric for examining execution efficiency alongside task quality in online GUI automation.
arXiv v1
OS-MAP: How Far Can Computer-Using Agents Go in Breadth and Depth?
OS-MAP organizes 416 computer-use tasks across 15 applications by a five-level automation taxonomy and a hierarchy of user demands. The benchmark separates required autonomy from generalization scope to diagnose where agents struggle as task demands increase.
arXiv v1
ProactiveVA: Proactive Visual Analytics with LLM-Based UI Agent
Studies proactive assistance in visual-analytics interfaces, using interaction logs to detect support needs and provide context-aware suggestions and guidance.
arXiv v1
MobileUse: A GUI Agent with Hierarchical Reflection for Autonomous Mobile Operation
MobileUse reflects on errors at action and task timescales, invoking reflection when needed and using proactive exploration to learn unfamiliar apps. It evaluates this mobile execution framework on AndroidWorld and AndroidLab and describes a toolkit for physical-device operation.
arXiv v3
GUI-G$^2$: Gaussian Reward Modeling for GUI Grounding
GUI-G2 replaces binary grounding rewards with Gaussian point rewards and spatial-coverage rewards. An adaptive variance tied to element dimensions supplies continuous training feedback for targets of different sizes, evaluated on the ScreenSpot benchmark family.
arXiv v1
Manipulating LLM Web Agents with Indirect Prompt Injection Attack via HTML Accessibility Tree
Studies indirect prompt injection through HTML content consumed as accessibility trees. The evaluation tests how manipulated page observations redirect browser-agent actions on real websites.
arXiv v5
MagicGUI: A Foundational Mobile GUI Agent with Scalable Data Pipeline and Reinforcement Fine-tuning
Combines a mobile GUI data pipeline, perception and planning capabilities, and reinforcement fine-tuning for interactive task execution.
arXiv v1
WebGuard: Building a Generalizable Guardrail for Web Agents
Introduces WebGuard, a human-annotated dataset for predicting outcomes and risk levels of state-changing website actions. It provides generalization splits and evaluates specialized guardrail models.
arXiv v2
Machine-Readable Ads: Accessibility and Trust Patterns for AI Web Agents interacting with Online Advertisements
Compares DOM-based and pixel-based web agents on a cloned news site containing advertisements, consent dialogs and paywalls. The study examines how interface presentation affects ad interaction and consequential subscription choices.
arXiv v3
LaSM: Layer-wise Scaling Mechanism for Defending Pop-up Attack on GUI Agents
Introduces LaSM, a training-free defense that selectively scales attention and MLP components to improve alignment with task-relevant screen regions. It evaluates the approach against pop-up-based GUI injection attacks.
arXiv v2
VisualTrap: A Stealthy Backdoor Attack on GUI Agents via Visual Grounding Manipulation
Studies backdoors that redirect GUI visual grounding even when the task plan is correct. VisualTrap evaluates whether poisoned grounding training transfers to downstream tasks and across web, mobile, and desktop interface settings.
arXiv v2
OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety
Evaluates agent safety across real browser, code, filesystem, shell, and messaging tools using rule-based and model-based assessments.
arXiv v5
GTA1: GUI Test-time Scaling Agent
GTA1 samples multiple action proposals at each GUI step and uses a judge to select among them. A separate grounding model is trained with reinforcement learning to map selected proposals to visual targets, linking planning-time search with click localization.
arXiv v1
MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environment
MobileGUI-RL constructs an online training curriculum through agent exploration and task filtering. It adapts GRPO with trajectory-aware advantages and rewards for task completion and execution efficiency, evaluated on three online mobile-agent benchmarks.
arXiv v1
R-VLM: Region-Aware Vision Language Model for Precise GUI Grounding
R-VLM localizes GUI elements through zoomed-in region proposals and an intersection-over-union-aware training objective. It evaluates the grounding model on ScreenSpot and AgentStudio and studies its effect on navigation in AITW and Mind2Web.
arXiv v1
A Systematization of Security Vulnerabilities in Computer Use Agents
Analyzes security boundaries in computer-use agents through a risk taxonomy and concrete adversarial case studies. It links observed failures to input provenance, interface-action binding, memory, and delegation, and proposes CUA-specific evaluation principles.
arXiv v2
Mobile GUI Agents under Real-world Threats: Are We There Yet?
Introduces an app-content instrumentation framework and complementary interactive and static test suites for mobile GUI agents. It studies how untrusted third-party content changes task execution and state understanding.
arXiv v4
How to Train Your LLM Web Agent: A Statistical Diagnosis
Studies compute allocation for web-agent post-training with teacher-generated supervised fine-tuning followed by on-policy reinforcement learning. It samples 1,370 training configurations and uses bootstrap analysis to examine hyperparameter sensitivity on WorkArena and MiniWoB++.
arXiv v1
Less is More: Empowering GUI Agent with Context-Aware Simplification
SimpAgent prunes distracting interface elements with visual masks and compresses interaction history using consistency-based guidance. The study evaluates these two forms of context reduction during GUI navigation in web and mobile environments.
arXiv v4
WebArxiv: A Reproducible Benchmark for Evaluating Multimodal Web Agents on arXiv Tasks
WebArxiv provides 510 static-snapshot tasks with deterministic answers for scholarly browsing, including constrained paper retrieval, content extraction, and cross-paper comparison. The study also tests dynamic retrieval of interaction history to address repetitive or incomplete agent reasoning.
arXiv v1
SafeMobile: Chain-level Jailbreak Detection and Automated Evaluation for Multimodal Mobile Agents
Studies chain-level risk detection for mobile multimodal agents and a language-model-assisted evaluator, assessing safety interventions together with functional preservation.
arXiv v1
LineRetriever: Planning-Aware Observation Reduction for Web Agents
LineRetriever uses a language model to select observation lines that matter for upcoming web-navigation actions. It reduces DOM or accessibility-tree context by considering the planning horizon instead of relying solely on semantic similarity.
arXiv v3
ZonUI-3B: A Lightweight Vision-Language Model for Cross-Resolution GUI Grounding
ZonUI-3B trains GUI grounding with a curated 24,000-example collection of mobile, desktop, and web screenshots. Its two-stage procedure first learns across platforms and then specializes on high-resolution images, with ablations on data diversity and redundant examples.
arXiv v4
Mobile-R1: Towards Interactive Capability for VLM-Based Mobile Agent via Systematic Training
Uses a staged curriculum from response-format alignment through action feedback to interactive task-level training for mobile agents.
arXiv v1
Chain-of-Memory: Enhancing GUI Agents for Cross-Application Navigation
Chain-of-Memory explicitly records action descriptions and task-relevant screen information in short- and long-term memory for GUI navigation. GUI Odyssey-CoM provides 111,000 screen-action pairs with memory annotations for learning and evaluating information retention across applications.
arXiv v2
Learning, Reasoning, Refinement: A Framework for Kahneman's Dual-System Intelligence in GUI Agents
CogniGUI combines hierarchical visual parsing of interface elements with a GRPO-based grounding agent that compares alternative interaction paths. Its ScreenSeek benchmark evaluates navigation across applications, changing interface states, and consistency between interfaces.
arXiv v2
OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agents
OSWorld-Human adds human-determined trajectories to OSWorld to evaluate the efficiency of desktop computer-use agents. The study compares 16 agents and analyzes model-call latency, changes in step duration during task execution, and excess action steps.
arXiv v1
Context manipulation attacks : Web agents are susceptible to corrupted memory
Studies context manipulation in web agents by altering stored task representations rather than webpage prompts alone. Evaluation on Browser-use and Agent-E examines memory-directed attacks and the implications for secure context handling.
arXiv v3
Embodied Web Agents: Bridging Physical-Digital Realms for Integrated Agent Intelligence
Introduces environments that couple 3D physical-world simulations with functional web interfaces. The accompanying benchmark evaluates tasks requiring agents to coordinate online information and physical actions across cooking, navigation and shopping scenarios.
arXiv v1
Understanding GUI Agent Localization Biases through Logit Sharpness
Introduces a finer-grained taxonomy of GUI localization errors and a logit-based Peak Sharpness Score for analyzing coordinate-prediction uncertainty. A context-aware cropping method refines the visual input without retraining the grounding model.
arXiv v2
OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents
Extends OSWorld with tasks covering user misuse, prompt injection, and model misbehavior, with assessment of both task success and safety.
arXiv v1
GUI-Robust: A Comprehensive Dataset for Testing GUI Agent Robustness in Real-World Anomalies
Builds GUI task data with real-world interaction anomalies and evaluates how agent behavior changes under those conditions.
arXiv v2
AgentSynth: Scalable Task Generation for Generalist Computer-Use Agents
AgentSynth composes easily generated subtasks into longer computer-use tasks and trajectories with adjustable difficulty. Its evaluation uses the number of subtasks to probe how agent success changes as task complexity grows.
arXiv v6
Poison Once, Control Anywhere: Clean-Text Visual Backdoors in VLM-based Mobile Agents
Demonstrates clean-text visual backdoor attacks (VIBMA) that compromise fine-tuned VLM mobile agents through visual triggers while keeping text instructions intact.
arXiv v2
DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning
Separates textual and iconic UI elements and iteratively focuses visual regions to refine grounding without additional model training.
arXiv v1
Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skills
Mirage-1 organizes interaction trajectories into execution, core, and meta-skills, then uses them to guide Monte Carlo tree search during online GUI tasks. It introduces AndroidLH for long-horizon evaluation and tests the approach on Android and web-navigation benchmarks.
arXiv v3
LPO: Towards Accurate GUI Agent Interaction via Location Preference Optimization
Location Preference Optimization uses an information-entropy signal to select interaction regions and a distance-based reward to assess click locations. It trains GUI localization through preference optimization and evaluates both offline grounding and online interaction.
arXiv v3
UITron-Speech: Towards Automated GUI Agents Based on Speech Instructions
Introduces UITron-Speech, which predicts GUI actions from spoken instructions and screenshots. It combines synthetic speech data and mixed-modality training with a training-free grounding refinement step.
arXiv v1
Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling System
UI-NEXUS evaluates compositional mobile tasks in local utility apps and online Chinese and English service apps. AGENT-NEXUS decomposes those long tasks into self-contained atomic subtasks to study generalization beyond isolated screens or short action chains.
arXiv v2
FingerTip 20K: A Benchmark for Proactive and Personalized Mobile LLM Agents
FingerTip 20K gathers 20K human Android demonstrations from users' long-term everyday app use, with user context, to evaluate proactive task suggestion and preference-aware task execution.
arXiv v3
EconWebArena: Benchmarking Autonomous Agents on Economic Tasks in Realistic Web Environments
Introduces EconWebArena, a curated benchmark of economic-data tasks requiring live website navigation, multimodal interpretation, and precise extraction. It evaluates visual grounding, planning, and interaction design within multistep workflows.
第 721–780 篇，共 1,200 篇，按发表时间由新到旧排列。