Source-backed papers on GUI grounding, agents, training, evaluation, and safety.
Read the research survey ↗ · Methods, evaluation, and human agency — an English working manuscript with a source-by-source reading register.
AI-assisted editorial review of primary sources; experiments are not independently reproduced. Dates keep their recorded precision. Resource links below follow reviewed introduction or documentation relationships. Method ↗
Surveys virtual agents for browsers, applications and operating systems, organizing prior work by task type, observation and action spaces, model strategies and evaluation methods. It examines screen-based interaction and keyboard, mouse and touch control, alongside transferability and long-sequence decision-making challenges.
Presents a preliminary study of Claude 3.5 Computer Use across desktop tasks and software, analyzing planning, action, and critique behavior and providing a framework for deploying GUI-automation models.
Introduces WebDreamer, which simulates candidate web actions with language-model world models and value estimates before acting.
Survey of (M)LLM-based GUI agents organized around their datasets and benchmarks, a unified agent framework with a taxonomy, and commercial applications.
Introduces an orchestrated team of browsing, file, and coding agents with progress tracking and error recovery for general digital tasks.
Introduces StepAgent, which compares agent and expert actions to supply intermediate rewards for policy learning, with implicit-reward and inverse-RL variants. Evaluation includes WebShop and Mind2Web alongside non-web tasks.
Tests whether adversarial visual pop-ups divert computer agents from intended tasks in desktop and web benchmark environments.
Trains web agents with an evolving task curriculum, outcome supervision, and reinforcement-learning updates intended to manage policy drift.
Surveys multimodal mobile agents, organizing prompt-based and training-based approaches alongside evaluation benchmarks, supporting technologies, and open challenges in mobile task execution.
Provides a shared Android action environment, reproducible tasks, and instruction data for training and comparing language and multimodal agents.
Analyzes how interaction history and webpage representation affect multiturn web-agent generalization across unfamiliar websites, task categories, and geographic settings.
Introduces CASA to assess web agents' responses to cultural and social norms in shopping and forum tasks, including misleading observations.
Findings of the Association for Computational Linguistics: NAACL 2025 ↗
Combines cross-platform GUI grounding data synthesis with action-model training to study generalization to unfamiliar interfaces.
Discovers compact intents from demonstrations and uses intent predictions to guide a language-model web agent's decisions and exploration.
Presents AutoGLM agents for browser and phone control, separating planning from grounding and training through an evolving interaction curriculum.
Develops multimodal browser agents through repeated cycles of real-web exploration, trajectory feedback, and policy optimization.
Synthesizes GUI training examples at multiple levels of detail from webpages and studies transfer of learned understanding to other interface environments.
Evaluates whether web agents can retain procedural skills and factual information from long video tutorials and use them during tasks.
Extends Ferret-UI across device platforms with adaptive visual scaling and synthetic data for referring, grounding, and user-centered UI tasks.
OSCAR translates user instructions into Python control code while observing screen images and acting through mouse and keyboard operations. A state-machine controller handles errors and replans tasks during execution across desktop and mobile applications.
Introduces a platform that integrates specialized agents under a coordinating MetaAgent for varied computer tasks.
LiMAC combines a compact Action Transformer with a fine-tuned vision-language model for Android app control. It uses textual goals and histories of screenshots and UI trees to predict actions, with evaluation on two mobile-control datasets.
Evaluates mobile agents on misuse, side effects, and indirect prompt-injection risks in an Android emulator environment.
Introduces a black-box web-agent red-teaming framework that learns adversarial prompts from agent feedback to study security weaknesses.
Compares browsing-only, API-only, and hybrid web agents on WebArena, examining how access to application APIs changes task execution alongside browser interaction.
Findings of the Association for Computational Linguistics: ACL 2025 ↗
Introduces a smartphone-agent benchmark with English and Chinese tasks, real-time Android interaction, and completion and resource-use metrics.
Introduces asynchronous distributed reinforcement learning with centralized updates and decentralized interaction collection for device-control agents.
Studies how simplifying and aligning a web agent's observations and actions with language-model capabilities affects task completion.
Builds MultiUI by generating instructions from webpage accessibility trees and pairing them with screenshots for multimodal training. Evaluation examines web-interface understanding, including VisualWebBench and Mind2Web, and transfer to other text-rich visual tasks.
Combines adaptive mobile-task planning, reflection-based recovery, and multiple memory components, evaluated with MobBench and AndroidArena.
Uses a learned web world model to predict important state changes and compare candidate actions before executing them.
Introduces BrowserART and studies whether chat-model refusal behavior transfers to browser agents carrying out potentially harmful instructions.
Develops a taxonomy of mobile-action consequences and annotated interaction data to evaluate whether models recognize impactful or risky UI operations.
Combines hierarchical planning, retrieved experience, and a computer interaction interface to execute multistep GUI tasks.
Separates multimodal reasoning and action planning from a dedicated UI localization model, with evaluation on Android devices.
Proceedings of the 26th Annual Meeting of the Special Interest Group on Discourse and Dialogue ↗
Adapts a compact Florence-2 model to map user instructions to screen coordinates using multitask training and augmented data.
Evaluates web-task completion under explicit enterprise policies, measuring policy-compliant completion and safety violations separately.
Introduces UGround, a visual target-localization model trained with large-scale synthetic references to elements in GUI screenshots.
Generates browser-agent demonstrations through autonomous exploration and retrospective task labeling, pruning trajectories that lack meaningful subtasks.
Studies text and visual validation for web agents and uses the resulting validator to detect failures and guide workflow self-refinement.
Dynamic Planning of Thoughts revises a mobile GUI plan using current environmental feedback and execution history. The study compares this planning strategy with long-dialogue prompting and examines its behavior on unfamiliar tasks and different language-model backbones.
Findings of the Association for Computational Linguistics: EMNLP 2024 ↗
Generates labeled slides and interface visuals through code to support visual element detection, description, and classification with less manual annotation.
Studies agent-system and interface co-design that incorporates user interruptions and intermediate feedback to reduce planning latency.
Introduces IconDesc, which generates mobile icon alt-text from partial UI context such as element identifiers, bounds, nearby text and hierarchy. Model evaluation and a user study examine the resulting accessibility descriptions.
Introduces AXIS, an API-first application agent that can use UI actions and expand its available APIs through automated application exploration. The work evaluates the approach on Microsoft Word tasks.
Synatra converts indirect procedural knowledge, including human-oriented tutorials, into direct demonstrations for digital agents. It trains a 7B CodeLlama on 100,000 synthesized demonstrations and evaluates the resulting policy on Mind2Web, MiniWoB++, and WebArena.
Introduces Steward, a natural-language browser automation system that repeatedly plans and executes website actions. The paper examines completion detection, state representation, responsiveness, and caching in real-site tasks.
MobileVLM adds two pretraining stages and four UI tasks to learn element detail and relationships between successive screens. Its Mobile3M dataset contains three million Chinese mobile UI pages connected by recorded transition actions.
Findings of the Association for Computational Linguistics: EMNLP 2024 ↗
Builds mobile screenshot and view-hierarchy pairs through automated application traversal and evaluates their usefulness for GUI grounding.
Maintains explicit task goals and evolving state descriptions to improve base-agent decisions across interactive tasks, including web shopping.
Studies how the ordering of UI elements in an agent observation affects web and desktop performance. A pixel-based element detector and dimensionality-reduction ordering are evaluated on OmniACT when document hierarchy is unavailable.
Evaluates privacy-directed webpage instruction injection against generalist web agents using PII-related interaction steps from Mind2Web, including attempts to expose specific user information or entire requests.
Orienter detects interactable GUI elements in XR applications by interpreting scene context and revisiting candidate elements through a validation and reflection loop. It addresses open-vocabulary element types and context-dependent interactability.
Frames web-application exploration as functionality-oriented question answering, generating navigation actions from multimodal context.
PeriGuru interprets smartphone GUI screenshots with computer vision and uses a language model to decide the next action. A peripheral robotic arm executes the phone interactions, providing an alternative mechanism for operating mobile apps.
Adapts real-desktop evaluation to Windows applications and supports parallel execution of benchmark tasks.
Agent Workflow Memory extracts reusable task routines and selectively supplies them to guide later web actions. It studies both offline workflow induction from training examples and online induction from test experience on Mind2Web and WebArena.
Introduces question answering over individual screens, multiple web pages, and navigation traces to test information synthesis across views.
Separates web-agent planning and grounding evaluations on Mind2Web to analyze which components limit task performance.
WebPilot separates global subtask planning from local Monte Carlo tree search during web execution. It continually refines the high-level plan and adapts local search to uncertain transitions and incomplete observations, evaluated on WebArena and MiniWoB++.
Showing 961–1,020 of 1,200, newest first.
