arXiv v2
GUIDE: Graphical User Interface Data for Execution
Introduces annotated website task trajectories with screenshots, prior actions, reasoning, and grounded next actions for process automation.
RESEARCH LIBRARY
Computer-use research library
Source-backed papers on GUI grounding, agents, training, evaluation, and safety.
Read the research survey ↗ · Methods, evaluation, and human agency — an English working manuscript with a source-by-source reading register.
PAPER TOPICSEvaluationData generationGUI groundingWeb navigationPlanningMobile navigationReinforcement learningSafety & securityAll topics ↗
AI-assisted editorial review of primary sources; experiments are not independently reproduced. Dates keep their recorded precision. Resource links below follow reviewed introduction or documentation relationships. Method ↗
arXiv v3
Autonomous Evaluation and Refinement of Digital Agents
Compares automatic trajectory evaluators for web navigation and device control against oracle judgments. The resulting feedback is used for agent fine-tuning and inference-time guidance, including experiments on WebArena.
arXiv v1
VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?
Separately evaluates web-page reading, understanding, and element grounding across tasks drawn from real websites.
arXiv v1
WILBUR: Adaptive In-Context Learning for Robust and Accurate Web Agents
WILBUR learns to rank prior task demonstrations for a web agent’s prompt and synthesizes instructions from past runs. A generated task curriculum and a backtracking mechanism support adaptation and recovery, evaluated on WebVoyager.
arXiv v1
Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs
Adapts multimodal modeling to mobile interfaces through detailed screen crops, region annotations, and UI reasoning tasks.
arXiv v2
AutoWebGLM: A Large Language Model-based Web Navigating Agent
Builds a web agent through simplified HTML observations, human-and-AI browsing data, curriculum training, and reinforcement-learning refinement.
arXiv v2
Prompt2Task: Automating UI Tasks on Smartphones from Textual Prompts
Prompt2Task converts textual goals or procedures into smartphone UI automation using agents for intent interpretation, external-information handling, and execution. User feedback and accumulated task knowledge support subsequent task generation and operation.
arXiv v2
ReALM: Reference Resolution As Language Modeling
Frames conversational and on-screen reference resolution as language modeling, allowing textual representations of screen entities to support user-reference interpretation. Evaluation separately examines on-screen references against an existing resolver and language-model baselines.
arXiv v3
AgentStudio: A Toolkit for Building General Virtual Agents
AgentStudio combines an interactive GUI/API environment with tools for creating tasks, annotating interface elements, and labeling actions in videos. Its online suite and GroundUI, IDMBench, and CriticBench datasets examine interaction, grounding, learning from video, and success detection.
arXiv v4
Tur[k]ingBench: A Challenge Benchmark for Web Agents
Evaluates web agents on multimodal annotation tasks adapted from HTML interfaces originally built for crowdsourcing workers.
arXiv v2
Computer User Interface Understanding. A New Dataset and a Learning Framework
Introduces desktop interaction videos and a framework combining synthetic examples with hierarchical contrastive learning for interface classification.
arXiv v5
WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?
Introduces enterprise web tasks on ServiceNow and BrowserGym interfaces for evaluating browser-based agents.
arXiv v1
UI Semantic Group Detection: Grouping UI Elements with Similar Semantics in Mobile Graphical User Interface
UISCGD detects semantic groups of adjacent text and non-text UI elements. It learns from Android and iOS screenshots to support interface retrieval, UI-to-code structure and screen-reader accessibility.
arXiv v1
PPTC-R benchmark: Towards Evaluating the Robustness of Large Language Models for PowerPoint Task Completion
Benchmarks robustness of API-based PowerPoint completion under sentence-level, semantic, and multilingual instruction perturbations and changes to the available API set.
arXiv v3
Cradle: Empowering Foundation Agents Towards General Computer Control
Uses screenshot observations, planning modules, and generated low-level keyboard and mouse control for general computer interaction.
Proceedings of the 42nd International Conference on Machine Learning ↗
arXiv v2
Android in the Zoo: Chain-of-Action-Thought for GUI Agents
Introduces Chain-of-Action-Thought and Android-In-The-Zoo annotations that connect prior actions, screen observations, intended actions, and expected outcomes.
Findings of the Association for Computational Linguistics: EMNLP 2024 ↗
arXiv v3
OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web
Introduces a dataset and benchmark for generating executable task scripts from screenshots and visually grounded desktop or web instructions.
arXiv v1
WIPI: A New Web Threat for LLM-Driven Web Agents
Studies WIPI, a black-box indirect-instruction threat in which webpage content redirects LLM-driven web agents. Evaluation spans plugin-based agents, Web GPTs, and open-source web-agent implementations.
arXiv v1
On the Multi-turn Instruction Following for Conversational Web Agents
MT-Mind2Web evaluates web navigation through sequences of conversational user instructions. Self-MAP combines memory-augmented planning with reflection to preserve context and dependencies across those interaction turns.
arXiv v1
Improving Language Understanding from Screenshots
Studies language understanding from text-rendered screenshots using joint image-patch and text prediction; a perception foundation relevant to screen-reading models.
Screen understandingAdjacent research
arXiv v3
CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation
Combines screenshots, layout information, and action history with a factored action predictor for smartphone automation.
Findings of the Association for Computational Linguistics: ACL 2024 ↗
arXiv v1Survey
A Trembling House of Cards? Mapping Adversarial Attacks against Language Agents
Presents a conceptual perception–reasoning–action framework and 12 potential attack scenarios for language agents, including hypothetical web-navigation attacks.
arXiv v2
OS-Copilot: Towards Generalist Computer Agents with Self-Improvement
Provides an operating-system agent framework connecting applications, files, terminals, and web tools, with accumulated skills for subsequent tasks.
arXiv v1
ScreenAgent: A Vision Language Model-driven Computer Control Agent
Combines a screenshot-and-action dataset with a planning, acting, and reflecting loop for computer control through mouse and keyboard actions.
arXiv v1
Understanding the Weakness of Large Language Model Agents within a Complex Android Environment
Introduces AndroidArena and analyzes agent limitations in cross-application tasks, user constraints, exploration, and reflection.
arXiv v5
UFO: A UI-Focused Agent for Windows OS Interaction
UFO uses two agents to inspect Windows application interfaces and their control metadata, then grounds and executes user requests within and across applications. The study evaluates this UI-focused architecture on nine Windows applications.
arXiv v2
WebLINX: Real-World Website Navigation with Multi-Turn Dialogue
Collects expert demonstrations of conversational web navigation and studies retrieval-based element selection for processing large pages.
Proceedings of the 41st International Conference on Machine Learning ↗
arXiv v3
ScreenAI: A Vision-Language Model for UI and Infographics Understanding
Introduces ScreenAI and screen-annotation training that supports UI question answering, navigation, and summarization data generation.
arXiv v2
Dual-View Visual Contextualization for Web Navigation
Uses screenshot geometry and visual content to enrich HTML element representations for web navigation. Nearby elements contribute textual and visual context, with evaluation on cross-task, cross-website, and cross-domain Mind2Web splits.
arXiv v2
Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception
Uses visual perception tools and task decomposition to operate mobile applications without relying on app interface metadata.
arXiv v3
GPTVoiceTasker: Advancing Multi-step Mobile Task Efficiency Through Dynamic Interface Exploration and Learning
Introduces GPTVoiceTasker, a mobile assistant that interprets user commands, executes device interactions, and learns from prior commands to streamline later task execution. The work includes a user study of task efficiency.
arXiv v4
WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models
Combines multimodal web interaction with a real-website task collection and an automated evaluation protocol.
arXiv v2
VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks
Introduces visually grounded website tasks for evaluating multimodal agents that must interpret both images and text.
arXiv v2
AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents
Introduces AgentBoard, a multiturn agent evaluation framework with intermediate progress metrics and analysis tools. Its broader task suite includes WebShop and WebArena interactions with stepwise progress measurements.
arXiv v2
SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents
Trains a screenshot-driven GUI agent with grounding pretraining and introduces ScreenSpot for evaluating interface target localization.
arXiv v3
MobileAgent: enhancing mobile control via human-machine interaction and SOP integration
Combines operating-procedure context with user interaction to guide mobile task automation and handle personalized or sensitive requests.
arXiv v2
GPT-4V(ision) is a Generalist Web Agent, if Grounded
Introduces SeeAct and live-website evaluation, examining how visual planning and element grounding affect web-agent task completion.
Proceedings of the 41st International Conference on Machine Learning ↗
arXiv v1
AutoTask: Executing Arbitrary Voice Commands by Exploring and Learning from Mobile GUI
Executes voice-requested mobile tasks through GUI exploration, backtracking, and accumulated interaction experience.
arXiv v1
WebVLN: Vision-and-Language Navigation on Websites
WebVLN trains website navigation from question-based instructions using both rendered visual information and underlying HTML. It introduces WebVLN-v1 and a website-aware navigation network to study this digital variant of vision-and-language navigation.
arXiv v3
AppAgent: Multimodal Agents as Smartphone Users
Builds app knowledge through exploration or human demonstrations and uses it to guide multimodal smartphone interactions.
arXiv v2
ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation
Introduces Windows application tasks and an actor-critic agent framework for mouse-and-keyboard automation of desktop workflows.
arXiv v2
VisionTasker: Mobile Task Automation Using Vision Based UI Understanding and LLM Task Planning
VisionTasker converts screenshots into textual UI interpretations and presents one interface at a time to an LLM planner for action selection. It evaluates visual representations and execution of 147 tasks on an Android smartphone, including integration with programming by demonstration.
arXiv v3
CogAgent: A Visual Language Model for GUI Agents
Introduces a visual-language model with low- and high-resolution image encoders for screenshot understanding and GUI navigation.
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ↗
arXiv v1
Intelligent Virtual Assistants with LLM-based Process Automation
Presents LLMPA, a mobile process-automation architecture that decomposes requests, identifies interface elements, predicts actions and checks errors. Demonstrations evaluate multistep operations in Alipay.
arXiv v3
Explore, Select, Derive, and Recall: Augmenting LLM with Human-like Memory for Mobile Task Automation
Introduces MobileGPT, which organizes mobile-app experience into reusable subtasks using an explore, select, derive, and recall cycle. Its memory supports learning task procedures and adapting them to new contexts.
arXiv v3
Exposing Limitations of Language Model Agents in Sequential-Task Compositions on the Web
CompWoB adds 50 composed web-automation tasks to study generalization from simpler interactions. The paper compares prompted and fine-tuned agents, balances task training data, and probes sensitivity to changes in instruction order.
arXiv v1
GAIA: a benchmark for General AI Assistants
Introduces questions that test assistant reasoning, multimodal understanding, browsing, and tool use beyond screen-action execution alone.
arXiv v1
GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation
Studies MM-Navigator, a multimodal agent that interprets smartphone screenshots and predicts localized actions without task-specific training.
arXiv v2
PPTC Benchmark: Evaluating Large Language Models for PowerPoint Task Completion
Evaluates multi-turn PowerPoint creation and editing, scoring the resulting presentation files rather than requiring a fixed API sequence.
arXiv v2
WebWISE: Web Interface Control and Sequential Exploration with Large Language Models
WebWISE filters DOM elements into observations and generates short programs for click, scroll, and text-entry steps. It evaluates in-context learning from either one supplied example or an example collected during a successful zero-shot MiniWoB++ trial.
arXiv v1
InstructExcel: A Benchmark for Natural Language Instruction in Excel
Introduces InstructExcel, a benchmark for generating Excel OfficeScripts from natural-language requests, built using scripts recorded from user actions and a collection of existing spreadsheets.
arXiv v2
AllTogether: Investigating the Efficacy of Spliced Prompt for Web Navigation using Large Language Models
AllTogether standardizes prompts for HTML-based web navigation and compares prompting with instruction fine-tuning. Its analysis examines how HTML context, interaction history, and current environmental feedback affect task execution.
arXiv v2
Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V
Overlays labeled image regions to help multimodal models refer to and reason about visual targets; a general visual-prompting method relevant to GUI grounding.
GUI groundingAdjacent research
arXiv v1
OpenAgents: An Open Platform for Language Agents in the Wild
OpenAgents hosts three language agents, for data analysis, API plugins and autonomous web browsing, behind a web interface for non-expert users, with local deployment for developers.
arXiv v3
A Zero-Shot Language Agent for Computer Control with Structured Reflection
Studies a zero-shot computer-control agent that plans executable actions and organizes its own reflections after mistakes. MiniWoB++ experiments examine learning to complete tasks without supplied expert trajectories.
arXiv v1
UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model
Introduces UReader, which combines unified instruction tuning, auxiliary text-reading tasks and shape-adaptive image crops for OCR-free visual language understanding. Its evaluation includes webpage screenshots alongside documents, tables, charts and natural images.
arXiv v1
ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations
ILuvUI generates 335,000 conversational training examples paired with UI screenshots, covering questions, descriptions, and plans. The resulting vision-language model is assessed on element detection, response quality, and examples of multi-step UI navigation.
arXiv v1
Reinforced UI Instruction Grounding: Towards a Generic UI Task Automation API
Trains a screenshot-based instruction-grounding model to predict target coordinates, using reinforcement learning to improve spatial decoding.
arXiv v4
SteP: Stacked LLM Policies for Web Actions
SteP dynamically composes specialized web-action policies through a stack that records the active chain of policy calls. It studies this control structure across WebArena, MiniWoB++, and CRM workflows.
arXiv v4
You Only Look at Screens: Multimodal Chain-of-Action Agents
Introduces Auto-GUI, a screenshot-based agent using previous actions and future action plans to guide device interaction.
Findings of the Association for Computational Linguistics: ACL 2024 ↗
Showing 1,081–1,140 of 1,200, newest first.