arXiv v2
SpiritSight Agent: Advanced GUI Agent with One Look
Introduces SpiritSight and GUI-Lasagne data, combining multilevel GUI annotations with visual block parsing for high-resolution interface navigation.
RESEARCH LIBRARY
Computer-Use 论文库
从 GUI 定位到智能体训练、评测与安全，每篇都可追溯到原始论文。
阅读英文研究综述 ↗ · 方法、评测与人的控制权；英文工作稿附有逐篇阅读记录。
条目经过 AI 辅助的一手来源编辑审阅，实验未独立复现。发表日期保留来源记录的精度。下方资源链接来自已审阅的介绍或文档关系。 收录方法 ↗
arXiv v2
LiteWebAgent: The Open-Source Suite for VLM-Based Web-Agent Applications
Introduces a modular visual web-agent stack with planning and memory components, exposed through a remote-browser application and Chrome extension.
arXiv v2
MPO: Boosting LLM Agents with Meta Plan Optimization
Refines reusable high-level meta plans with execution feedback to guide language-agent planning across interactive environments.
arXiv v2
AutoEval: A Practical Framework for Autonomous Evaluation of Mobile Agents
Generates reward signals from UI-state changes and uses a judging system to evaluate mobile agents with reduced manual evaluator construction.
arXiv v3
AppAgentX: Evolving GUI Agents as Proficient Smartphone Users
Records interaction history and converts repeated action sequences into reusable high-level shortcuts for mobile GUI tasks.
arXiv v2
ATLaS: Agent Tuning via Learning Critical Steps
Selects critical steps from expert interaction trajectories for targeted agent fine-tuning, studying efficiency and generalization across tasks.
arXiv v2
Smoothing Grounding and Reasoning for MLLM-Powered GUI Agents with Query-Oriented Pivot Tasks
Uses inference of possible user queries from screenshots and target coordinates as a training bridge between GUI grounding and action reasoning.
arXiv v3
Why Are Web AI Agents More Vulnerable Than Standalone LLMs? A Security Analysis
Analyzes web-agent vulnerabilities at the component level, examining goal placement in prompts, multistep action generation, and observations as sources of increased susceptibility.
arXiv v3
OS-Kairos: Adaptive Interaction for MLLM-Powered GUI Agents
Uses step-level confidence estimates to decide when a GUI agent should act autonomously or request human intervention in complex situations.
arXiv v1
VEM: Environment-Free Exploration for Training GUI Agent with Value Environment Model
Trains GUI agents with a frozen value model learned from offline interactions, guiding action exploration without predicting complete next observations.
arXiv v1
Towards Sustainable Web Agents: A Plea for Transparency and Dedicated Metrics for Energy Consumption
Examines energy and carbon-accounting limitations in web-agent evaluation and argues for transparent, dedicated sustainability measurements.
arXiv v3
Towards Enterprise-Ready Computer Using Generalist Agent
Describes the CUGA enterprise-agent development process, emphasizing iterative evaluation, failure analysis, and refinement on software-task benchmarks.
arXiv v2
Programming with Pixels: Can Computer-Use Agents do Software Engineering?
Introduces a visual IDE environment and benchmark for software-engineering tasks, comparing screenshot-based control with access to editing and shell tools.
arXiv v3
Mobile-Agent-V: A Video-Guided Approach for Effortless and Efficient Operational Knowledge Injection in Mobile Automation
Extracts operational guidance from video for mobile task execution and introduces Mobile-Knowledge to evaluate the contribution of external knowledge.
arXiv v2
PC-Agent: A Hierarchical Multi-Agent Collaboration Framework for Complex Task Automation on PC
Uses active perception and hierarchical agents for instruction decomposition, progress tracking, action selection, and reflection in PC workflows.
arXiv v3
Evaluating the Robustness of Multimodal Agents Against Active Environmental Injection Attacks
Examines active environmental injection against mobile agents, testing whether adversarial interface content redirects decisions during task execution.
arXiv v4
Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agents
Synthesizes multimodal web trajectories through exploration and refinement, then trains a browser agent to study the effect of scaling interaction data.
Findings of the Association for Computational Linguistics: ACL 2025 ↗
arXiv v1
Digi-Q: Learning Q-Value Functions for Training Device-Control Agents
Learns action-value functions from offline device-control experience and uses them to rank candidate actions for policy improvement.
arXiv v2
TRISHUL: Towards Region Identification and Screen Hierarchy Understanding for Large VLM based GUI Agents
Combines hierarchical screen parsing with spatially enriched element descriptions to support both locating GUI targets and describing indicated elements.
arXiv v5
WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Point
Introduces desktop and web tasks with varied initial states to test planning recovery and adaptability beyond canonical starting configurations.
arXiv v2
Advancing Autonomous VLM Agents via Variational Subgoal-Conditioned Reinforcement Learning
Uses variational subgoal-conditioned reinforcement learning to improve learning efficiency on sparse-reward mobile and web-control tasks.
arXiv v2
Symbiotic Cooperation for Web Agents: Harnessing Complementary Strengths of Large and Small LLMs
Couples large- and small-model web-agent exploration with trajectory synthesis and distillation to improve both model types iteratively.
arXiv v1
AppVLM: A Lightweight Vision Language Model for Online App Control
AppVLM first learns app actions from AndroidControl and then refines its policy through data collected in AndroidWorld. The study compares offline action prediction and online task completion for a lightweight vision-language smartphone agent.
arXiv v2
MobileA3gent: Training Mobile GUI Agents Using Decentralized Self-Sourced Data from Diverse Users
Studies automatic annotation and federated training of mobile GUI agents from decentralized user interaction data with heterogeneous distributions.
arXiv v1
ReachAgent: Enhancing Mobile Agent via Page Reaching and Operation
Splits mobile tasks into reaching the required page and operating on it, using subtask data and preferred GUI trajectories for training.
arXiv v2
AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMs
Infers element functionality from UI changes before and after interaction to produce verified annotations for grounding-model training.
arXiv v1
Towards Computer-Using Personal Agents
Proposes computer-using personal agents connected to user-controlled data repositories, discussing privacy, interoperability, and collaborative tasks.
arXiv v4
CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web Navigation
Introduces CowPilot, a web-navigation framework where agents suggest actions and users can pause, reject, override, or resume them. It evaluates autonomous and collaborative use through task success and human effort.
arXiv v1
WinClick: GUI Grounding with Multimodal Large Language Models
Introduces a screenshot-based Windows GUI agent and the WinSpot benchmark for evaluating localization of actionable interface regions.
arXiv v3综述
A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directions
Reviews 87 computer-use agents and 33 datasets through a taxonomy of operating domain, observation and action modalities, and agent design.
跨平台综述
arXiv v2
GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration
Collects environment-specific interaction data through autonomous exploration to adapt grounding models to previously unseen GUIs.
arXiv v1
UI-TARS: Pioneering Automated GUI Interaction with Native Agents
Studies a screenshot-driven native GUI model with unified actions, deliberate reasoning, and iterative training on interaction traces.
arXiv v2
Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks
Uses hierarchical mobile agents and persistent tips and reusable action shortcuts to improve complex tasks from accumulated experience.
arXiv v3
Agent-R: Training Language Model Agents to Reflect via Iterative Self-Training
Builds self-critique training examples from searched recovery paths to improve agents' ability to detect and correct errors during interaction.
arXiv v3
WebWalker: Benchmarking LLMs in Web Traversal
Introduces WebWalkerQA and an exploration-and-critique framework for gathering information by traversing multiple levels of a website.
arXiv v1
InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection
Uses staged supervised training to combine GUI understanding and grounding with hierarchical reasoning and expectation-based reflection.
arXiv v3
A3: Android Agent Arena for Mobile GUI Agents with Essential-State Procedural Evaluation
Introduces Android Agent Arena with online application tasks and procedural evaluation based on reaching essential intermediate states.
arXiv v3
OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis
Builds GUI trajectories by exploring environments first and deriving tasks retrospectively, with reward-model filtering of generated trajectories.
跨平台数据生成
arXiv v1
GUI Testing Arena: A Unified Benchmark for Advancing Autonomous GUI Testing Agent
Evaluates GUI testing through test-intent generation, execution, and defect detection using real, modified, and synthetic mobile application data.
arXiv v3
AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generation
Recasts mobile UI automation as on-device code generation, using app-specific documentation and synthetic tasks to guide smaller models.
arXiv v1
PC Agent: While You Sleep, AI Works -- A Cognitive Journey into Digital World
Combines PC Tracker interaction capture, completion of action and reasoning annotations, and planning and grounding agents for desktop workflows.
arXiv v2
Aria-UI: Visual Grounding for GUI Instructions
Introduces a vision-based grounding model trained with diverse synthetic instructions and action histories for context-sensitive GUI targeting.
Findings of the Association for Computational Linguistics: ACL 2025 ↗
arXiv v3
TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks
Simulates a software-company workplace to evaluate agents on professional tasks involving websites, programs, files, and coworker communication.
arXiv v3综述
GUI Agents: A Survey
Categorizes GUI-agent benchmarks, evaluation metrics, architectures, and training methods, and proposes a unified perception, reasoning, planning, and acting framework.
跨平台综述
arXiv v1
Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agents
Combines autonomous task proposal, web interaction, and visual success evaluation to discover skills and improve browser-agent policies.
arXiv v1
Attention-driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models without Fine-Tuning
Locates GUI targets by identifying and aggregating attention maps in pretrained multimodal models without task-specific fine-tuning.
跨平台GUI 定位
arXiv v1
WEPO: Web Element Preference Optimization for LLM-based Web Navigation
Uses web-element preference optimization with sampled negative HTML elements to align high-level user intent with navigation actions.
arXiv v2
Iris: Breaking GUI Complexity with Adaptive Focus and Self-Refining
Combines information-sensitive image cropping with a learning loop between element description and grounding for complex interface understanding.
arXiv v2
Large Action Models: From Inception to Implementation
Uses a Windows-agent case study to describe a development workflow for large action models, covering data collection, training, grounding, environment integration, and evaluation.
arXiv v2
AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials
Converts public tutorials into task specifications and executes them with an agent to synthesize multimodal interaction trajectories.
arXiv v1
Falcon-UI: Understanding GUI Before Following User Instructions
Pretrains GUI-context understanding with instruction-free web interactions before fine-tuning Falcon-UI on instructed GUI tasks.
arXiv v4
The BrowserGym Ecosystem for Web Agent Research
Describes BrowserGym's shared web interaction environment and AgentLab tooling for consistent experiments across multiple agent benchmarks.
arXiv v2
Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction
Uses screenshots, a unified action space, and staged grounding and reasoning training for cross-platform GUI interaction.
arXiv v1
Ponder & Press: Advancing Visual GUI Agent towards General Computer Control
Separates interpretation of user instructions from precise GUI localization using general-purpose and specialized visual-language components.
Findings of the Association for Computational Linguistics: ACL 2025 ↗
arXiv v1
DroidCall: A Dataset for LLM-powered Android Intent Invocation
Introduces data and a generation pipeline for translating natural-language requests into Android intent invocations using compact language models.
arXiv v12综述
Large Language Model-Brained GUI Agents: A Survey
Survey of LLM-brained GUI agents that carry out natural-language instructions across web, mobile, and desktop interfaces, organizing research and industry progress.
跨平台综述
arXiv v1
ShowUI: One Vision-Language-Action Model for GUI Visual Agent
Combines visual token selection, interleaved visual-action history, and curated instruction data in a compact GUI action model.
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ↗
arXiv v2
ScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow Data
Studies fine-tuning open language models on production web-workflow traces, including training choices for specialized web agents.
arXiv v1
AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations
Adapts multimodal web agents to unfamiliar websites using a small number of human demonstrations through prompting or meta-learning.
arXiv v7
Improved GUI Grounding via Iterative Narrowing
Uses iterative visual narrowing to improve localization of requested GUI elements with general or specially trained vision-language models.
跨平台GUI 定位
第 901–960 篇，共 1,200 篇，按发表时间由新到旧排列。