arXiv v2
ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory
Distills generalizable reasoning strategies from both successful and failed agent trajectories into a persistent memory bank for test-time retrieval.
RESEARCH LIBRARY
Computer-Use 论文库
从 GUI 定位到智能体训练、评测与安全，每篇都可追溯到原始论文。
阅读英文研究综述 ↗ · 方法、评测与人的控制权；英文工作稿附有逐篇阅读记录。
条目经过 AI 辅助的一手来源编辑审阅，实验未独立复现。发表日期保留来源记录的精度。下方资源链接来自已审阅的介绍或文档关系。 收录方法 ↗
arXiv v1
Scaling Synthetic Task Generation for Agents via Exploration
AutoPlay explores application states and functions to generate tasks, then uses a model executor and verifier to construct demonstrations. It produces 20,000 tasks across 20 Android apps and 10,000 tasks across 13 Ubuntu apps for supervised and reinforcement learning of UI agents.
arXiv v1
Retrieval-augmented GUI Agents with Generative Guidelines
RAG-GUI trains a small guidance adapter to filter and summarize retrieved web tutorials while keeping the acting agent fixed. Supervised and rejection-sampling stages prepare the adapter for inference-time assistance, tested on offline web/mobile tasks and interactive Android tasks.
arXiv v1
Generalist Scanner Meets Specialist Locator: A Synergistic Coarse-to-Fine Framework for Robust GUI Grounding
GMS pairs a general vision-language scanner that proposes interface regions with a specialist locator that predicts coordinates within them. Hierarchical coarse-to-fine search coordinates the two components for GUI grounding on ScreenSpot-Pro.
arXiv v1
WAREX: Web Agent Reliability Evaluation on Existing Benchmarks
WAREX augments existing web-agent benchmarks with reliability challenges arising from unstable systems and changing or disruptive website behavior. Evaluation on WebArena, WebVoyager and REAL examines how these disturbances affect task completion.
arXiv v1
Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curation
DART separates GUI reinforcement learning into asynchronous environment, rollout, data-management, and training services. It curates experience by task difficulty, successful trajectories, and step uncertainty while correcting rollout-policy mismatch, with evaluation on OSWorld.
arXiv v3
GUI-PRA: Process Reward Agent for GUI Tasks
GUI-PRA evaluates GUI progress by generating task-specific criteria and actively inspecting visual evidence with tools. It tests offline judgment on OS-Critic Bench and the effects of this feedback on online mobile policies in AndroidWorld.
arXiv v4
Secure and Efficient Access Control for Computer-Use Agents via Context Space
Introduces CSAgent, which combines static access policies with user intent and context and enforces them through an operating-system service. A supporting toolchain constructs and refines policies for agents using API, CLI and GUI interfaces.
arXiv v2
RISK: A Framework for GUI Agents in E-commerce Risk Management
RISK combines browser-interaction training data, a benchmark, and reinforcement fine-tuning for e-commerce risk-management workflows. Its trajectories and rewards distinguish individual actions from multi-step stateful tasks involving deeply embedded web content.
arXiv v2
ProRe: A Proactive Reward System for GUI Agents via Reasoner-Actor Collaboration
ProRe evaluates GUI task outcomes by having a reasoner assign targeted environment-probing tasks to specialist actors. The additional observations support reward judgments, with separate evaluation of judge accuracy and downstream agent task success.
arXiv v2
Benchmarking MLLM-based Web Understanding: Reasoning, Robustness and Safety
Evaluates multimodal LLMs on web interface reasoning, visual perturbation robustness, and safety-critical web understanding tasks.
arXiv v2
Learning GUI Grounding with Spatial Reasoning from Visual Feedback
GUI-Cursor treats GUI grounding as a sequence of cursor movements guided by visible feedback and movement history. Online reinforcement learning with trajectory rewards trains the model to refine the target location across multiple steps.
arXiv v2
Robust, Observable, and Evolvable Agentic Systems Engineering: A Principled Framework Validated via the Fairy GUI Agent
Presents an agent-engineering framework combining runtime goal clarification, an observable component architecture and evolving execution memory. The Fairy mobile GUI agent evaluates these mechanisms on ambiguous tasks and through controlled and user studies.
arXiv v1
Orcust: Stepwise-Feedback Reinforcement Learning for GUI Agent
Orcust combines principle-constrained reward modeling with GUI trajectory collection in instrumented virtual machines. Environment-checkable and model-derived principles guide stepwise feedback for grounding, basic operations, and complete task execution.
arXiv v1
UIPro: Unleashing Superior Interaction Capability For GUI Agents
UIPro pretrains GUI understanding on 20.6 million tasks and then harmonizes interaction datasets through a unified action space. Continued fine-tuning uses the merged multi-platform data to train action prediction for generalist GUI operation.
arXiv v1
GUI-ReWalk: Massive Data Generation for GUI Agent via Stochastic Exploration and Intent-Aware Reasoning
GUI-ReWalk begins with stochastic interface exploration and then infers goals to generate coherent action trajectories. It supports long-horizon workflows across applications and studies training on the resulting data using grounding and navigation benchmarks.
arXiv v4
BTL-UI: Blink-Think-Link Reasoning Model for GUI Agent
BTL-UI separates GUI interaction into screen-region attention, planning and executable-command generation. It adds an automated annotation pipeline and rule-based process/outcome rewards for training, with evaluation on both static understanding and dynamic interaction.
arXiv v1
GUI-ARP: Enhancing Grounding with Adaptive Region Perception for GUI Agents
GUI-ARP uses visual attention to crop relevant screenshot regions and decides whether a target needs single-stage or multi-stage analysis. Supervised fine-tuning followed by GRPO trains this adaptive procedure for high-resolution GUI grounding.
arXiv v2
ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data
Introduces a cross-platform computer-use dataset and agents trained with a collection pipeline combining automated interaction and human input.
arXiv v1
Detecting Pipeline Failures through Fine-Grained Analysis of Web Agents
Introduces a modular framework for identifying errors at intermediate stages of a web-agent pipeline. A SeeAct and Mind2Web case study examines weaknesses that are obscured by final task-success metrics.
arXiv v2
TGPO: Tree-Guided Preference Optimization for Robust Web Agent Reinforcement Learning
TGPO merges semantically equivalent web states into tree-structured trajectories for offline preference optimization. A process reward model scores subgoal progress, redundant behavior, and action validity, while dynamic weighting emphasizes consequential decisions.
arXiv v1
InfraMind: A Novel Exploration-based GUI Agentic Framework for Mission-critical Industrial Management
InfraMind explores industrial management GUIs using virtual-machine snapshots and builds memory for planning and hierarchical state recognition. Its framework also studies knowledge distillation and safety checks on open-source and commercial data-center management platforms.
arXiv v5
See, Think, Act: Teaching Multimodal Agents to Effectively Interact with GUI by Identifying Toggles
Introduces a benchmark for binary GUI-toggle instructions and State-aware Reasoning, which compares the observed control state with the requested state before choosing an action. It evaluates agents on toggle tasks and broader interactive benchmarks.
arXiv v1
PrivWeb: Unobtrusive and Content-aware Privacy Protection For Web Agents
Introduces PrivWeb, which uses a local language model to anonymize interface information according to user preferences. Adaptive notifications and selective pauses give users control over sensitive information access during web-agent tasks.
arXiv v2
How Auxiliary Reasoning Unleashes GUI Grounding in VLMs
Studies the gap between latent visual grounding and explicit coordinate output in general vision-language models. Three zero-shot methods add axes, grids, or labeled intersections to GUI screenshots so models can express spatial locations without fine-tuning.
arXiv v2
UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learning
UI-S1 simulates multi-turn online learning over offline trajectories, using a patch module to recover divergences while retaining the model’s own outputs in context. It combines discounted future returns with step- and episode-level advantages and introduces a semi-online performance proxy.
arXiv v3
Environmental Injection Attacks against GUI Agents in Realistic Dynamic Environments
Introduces Chameleon, an environmental-injection evaluation framework that varies webpage content and trigger context. It combines simulated page variation with attention-based optimization and tests GUI agents across realistic websites.
arXiv v1
Dark Patterns Meet GUI Agents: LLM Agent Susceptibility to Manipulative Interfaces and the Role of Human Oversight
Studies how GUI agents, people, and human-agent teams respond to manipulative interface designs. The two-phase evaluation examines failure modes and the benefits and costs of human oversight.
arXiv v2
MobileRL: Online Agentic Reinforcement Learning for Mobile GUI Agents
MobileRL adapts GRPO to mobile GUI training with difficulty-dependent replay of successes and filtering of failure curricula. A shortest-path reward adjustment accounts for task length, with evaluations on AndroidWorld and AndroidLab.
arXiv v1
AgentSentinel: An End-to-End and Real-Time Security Defense Framework for Computer-Use Agents
Introduces AgentSentinel, which intercepts sensitive computer-agent operations and audits them using task context together with system traces. BadComputerUse evaluates this enforcement approach across attack scenarios involving tool and computer control.
arXiv v3
VeriOS: Query-Driven Proactive Human-Agent-GUI Interaction for Trustworthy OS Agents
Introduces VeriOS-Agent, which learns when to execute GUI actions autonomously and when to query a user under untrustworthy conditions. Its staged training combines supervised learning and group-relative policy optimization.
arXiv v2
MAS-Bench: A Unified Benchmark for Shortcut-Augmented Hybrid Mobile GUI Agents
MAS-Bench evaluates mobile agents that combine GUI actions with APIs, deep links, and reusable scripts. Its 139 tasks across 11 apps test use of 88 predefined shortcuts as well as the ability to generate new reusable workflows.
arXiv v1
SparkUI-Parser: Enhancing GUI Perception with Robust Grounding and Parsing
Introduces SparkUI-Parser, which adds continuous coordinate decoding and a rejection mechanism for nonexistent targets to a multimodal model. It also presents ScreenParse for evaluating structural interface perception.
arXiv v1
Learning Active Perception via Self-Evolving Preference Optimization for GUI Grounding
LASER builds preference data for multi-step GUI perception by combining Monte Carlo quality estimates with region-overlap evaluation. Its self-evolving training encourages relevant region selection and adapts the amount of reasoning before predicting screen coordinates.
arXiv v1
MobileRAG: Enhancing Mobile Agent with Retrieval-Augmented Generation
MobileRAG combines retrieval and memory components to support mobile tasks that need external knowledge or long action sequences. MobileRAG-Eval is introduced to study these requirements on complex real-world app tasks.
arXiv v2
UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning
UI-TARS-2 combines scalable trajectory generation, stabilized multi-turn reinforcement learning, and a sandbox that supports GUI, filesystem, and terminal actions. The report evaluates the resulting agent across web, desktop, Android, and game tasks and analyzes training stability.
arXiv v2
AppCopilot: Toward General, Accurate, Long-Horizon, and Efficient Mobile Agent
Describes a mobile-agent stack combining region-based coordinate correction, hierarchical task planning, multi-agent voting, and reuse of successful interaction traces.
arXiv v1
OmniActor: A Generalist GUI and Embodied Agent for 2D&3D Worlds
Introduces OmniActor, which shares shallow model layers while separating deeper parameters to train jointly on GUI and embodied interactions. A unified action representation and mixed training data support evaluation in both task families.
arXiv v2
Succeed or Learn Slowly: Sample Efficient Off-Policy Reinforcement Learning for Mobile App Control
Succeed or Learn Slowly applies direct off-policy updates to successful interactions and more conservative regularized updates to negative examples. Successful Transition Replay prioritizes successful experience during policy learning, with evaluation on AndroidWorld.
arXiv v1
Throttling Web Agents Using Reasoning Gates
Introduces reasoning gates that require agents to solve generated, verifiable puzzles before receiving access to resources. The throttling approach is deployed on a custom website and MCP servers and evaluated with web agents.
arXiv v2
FlashAdventure: A Benchmark for GUI Agents Solving Full Story Arcs in Diverse Adventure Games
FlashAdventure evaluates GUI agents on completing story arcs in 34 adventure games. It pairs an automated gameplay judge with COAST, a framework that keeps long-term clue memory to plan and complete sequential objectives.
arXiv v1
MobiAgent: A Systematic Framework for Customizable Mobile Agents
Presents MobiAgent, combining MobiMind agent models, the AgentRR acceleration framework, and MobiFlow evaluation. It also introduces an AI-assisted data-collection pipeline for mobile GUI tasks.
arXiv v1
KG-RAG: Enhancing GUI Agent Decision-Making via Knowledge Graph-Driven Retrieval-Augmented Generation
KG-RAG converts UI transition graphs into a searchable vector store and uses intent-guided retrieval to construct actionable navigation paths. It introduces KG-Android-Bench and KG-Harmony-Bench and evaluates transfer from mobile apps to web and desktop tasks.
arXiv v1
UItron: Foundational GUI Agent with Advanced Perception and Planning
UItron combines GUI perception and planning data engineering with supervised training and curriculum reinforcement learning in connected mobile and PC environments. It also collects more than one million operation steps across 100 Chinese apps and builds offline and online evaluation environments.
arXiv v1
Morae: Proactively Pausing UI Agents for User Choices
Introduces Morae, a UI agent that detects decision points and pauses for user choices using interface code and screenshots. A study with blind and low-vision participants evaluates this mixed-initiative approach on real web tasks.
arXiv v1
CODA: Coordinating the Cerebrum and Cerebellum for a Dual-Brain Computer Use Agent with Decoupled Reinforcement Learning
CODA combines a generalist planner with a specialist GUI executor and trains application-specific planners through decoupled GRPO. Successful specialist trajectories then support supervised training of a consolidated planner, evaluated across four ScienceBoard applications.
arXiv v2
InquireMobile: Teaching VLM-based Mobile Agent to Request Human Assistance via Reinforcement Fine-Tuning
InquireMobile trains mobile agents to seek human confirmation at critical decisions using staged reinforcement fine-tuning and pre-action reasoning. InquireBench evaluates safe interaction and proactive inquiry, distinguishing asking for help from completing the underlying task.
arXiv v2
Mind the Third Eye! Benchmarking Privacy Awareness in MLLM-powered Smartphone Agents
Introduces SAPA-Bench with 7,138 smartphone scenarios to evaluate whether MLLM-powered mobile agents detect and protect sensitive personal information on screen.
arXiv v2
AppAgent-Pro: A Proactive GUI Agent System for Multidomain Information Integration and User Assistance
Proposes a proactive GUI assistant that gathers information across domains in response to user requests and anticipated information needs.
arXiv v1
PerPilot: Personalizing VLM-based Mobile Agents via Memory and Exploration
Introduces PerPilot and the PerInstruct dataset for completing user-specific mobile instructions through memory retrieval, app exploration, and clarification when missing context remains unresolved.
arXiv v1
WebSight: A Vision-First Architecture for Robust Web Agents
WebSight uses a vision-language model trained on web-interface examples within a modular agent with planning, action, verification, and episodic memory components. It evaluates element clicking on Showdown Clicks and full browsing tasks on WebVoyager using visual observations.
arXiv v2
Mobile-Agent-v3: Fundamental Agents for GUI Automation
Introduces GUI-Owl models and Mobile-Agent-v3 orchestration, using environment-generated trajectories to support GUI capabilities.
arXiv v2
ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agents
Combines API and GUI actions with distributed desktop reinforcement learning and alternating supervised training to address training instability.
arXiv v3
V2P: Visual Attention Calibration for GUI Grounding via Background Suppression and Center Peaking
V2P trains GUI grounding by suppressing attention to background regions and emphasizing the center of the target element. It represents target importance with size-dependent Gaussian heatmaps and evaluates the contribution of each component on ScreenSpot-v2 and ScreenSpot-Pro.
arXiv v2
A Functionality-Grounded Benchmark for Evaluating Web Agents in E-commerce Domains
Amazon-Bench generates functionality-specific web tasks from page content and interactive controls, covering operations beyond product search. Its evaluator checks task completion and unintended changes to account state, such as address or purchase settings.
arXiv v3
WebMall -- A Multi-Shop Benchmark for Evaluating Web Agents
WebMall provides four simulated online shops with heterogeneous product data for comparison shopping, related-product searches, and checkout tasks. It evaluates web agents with different observation formats and memory configurations in a fixed offline environment.
arXiv v1
You Don't Know Until You Click:Automated GUI Testing for Production-Ready Software Evaluation
RealDevWorld evaluates generated software repositories with 194 development tasks and AppEvalPilot, an agent judge that operates the resulting GUIs. The evaluation examines functional behavior, visual fidelity, and runtime responses to produce task-specific diagnostic feedback.
arXiv v2
CRAFT-GUI: Curriculum-Reinforced Agent For GUI Tasks
CRAFT-GUI organizes GUI trajectories by difficulty for curriculum-based GRPO training. Its reward combines rule-based signals with model judgments, with evaluation on Android Control and an internal online benchmark.
arXiv v2
UI-Venus Technical Report: Building High-performance UI Agents with RFT
Studies screenshot-based GUI grounding and navigation using curated interaction data, reward design, and reinforcement fine-tuning.
arXiv v1
FineState-Bench: A Comprehensive Benchmark for Fine-Grained State Control in GUI Agents
FineState-Bench evaluates fine-grained GUI perception and control across desktop, web, and mobile interfaces. Its Visual Diagnostic Assistant separates perception and localization contributions in controlled interaction experiments.
第 661–720 篇，共 1,200 篇，按发表时间由新到旧排列。