Interaction-as-a-Judge
Uses computer-use agents to interact with generated applications, studying whether interaction improves preference judging, requirement checks, and feedback for application generation.
WebDesktop
Automation
Source-reviewed research and assets tagged with Automation.
Membership follows reviewed tags; it does not establish a dependency between neighboring records. Dates preserve source precision. Annotations retain their original language.
When to Compile a Computer-Use Agent? Measuring Payback and Making Compilation Decisions for Token Efficiency
Introduces PACE, a protocol and online decision rule for deciding when repeated GUI procedures are worth compiling into programs. It accounts for failed compilation attempts, measured per-use savings, uncertain reuse, and execution costs under an explicit cumulative budget.
GUI-HARVEST: Self-Improving GUI Agents through Evidence-Driven Harness Evolution
Improves GUI-agent execution code by analyzing action effects and repeated-task failures, then validating bounded changes with frozen model weights.
DesktopWindows
Neuro-Symbolic Computer Use: Learning Reusable Policies for Reliable and Efficient Execution
Learns reusable computer-use policies that encode stable workflow logic in programs while delegating grounding and state checks to neural models. Policy iteration executes variants, diagnoses failures, revises the program, and adds verification before state-changing actions.
Desktop
PhoneCLI: From App Interfaces to Callable Commands for Mobile Agents
Introduces PhoneCLI, which explores mobile app interfaces and compiles navigation paths into reusable commands without internal app APIs or instrumentation. At execution time it verifies compiled paths and falls back to a visual agent for open-ended interaction or failed navigation.
Android
EchoPath: Execution-Level Replayable Memory for GUI Agents
EchoPath turns validated GUI trajectories into parameterized replay procedures with explicit runtime preconditions. Image-based target matching adjusts stored coordinates, while ambiguous steps fall back to bounded repair or fresh planning.
Improving Proficiency and Efficiency of Android GUI Agents via Self-Generating Tool Actions
DroidTool generates application-state tools through proposal, implementation, testing and repair. Relational tests connect the tools’ preconditions and effects, and the resulting tools are evaluated as additions to Android GUI agents.
Android
OmegaUse-SOP: SOP Engineering for Professional Computer Use from Human Demonstrations
OmegaUse-SOP converts expert GUI demonstrations into configurable procedures and refines them with domain rules and human input. It records, abstracts, grounds and verifies live execution, with a case study of photovoltaic simulation workflows in PVsyst 7.2.
SCAFFOLD: Self-Improving Web Agents via Recursive Parametric Skill Abstraction
Introduces SCAFFOLD, which extracts executable skills from web trajectories, composes them into a hierarchy, and compresses redundant skills through behavioral checks. It periodically distills skill-guided trajectories into the policy and studies repeated improvement across web benchmarks.
Web
UI-Venus-2 Technical Report
Describes a GUI foundation agent trained through expanded environments, function-grounded tasks, and trajectory verification with safety-aware execution mechanisms.
WebDesktopCross-platform
ASIL: Replacing Screenshot-and-Click with Structured State and Semantic Actions
ASIL exposes application state as structured JSON and provides code-executable semantic actions. Its application benchmark compares this interface with screenshot control and application-native APIs, and studies supervised and reinforcement learning on the structured interface.
MAGA: Multi-Platform Self-Fusion of GUI Agents via Structured Action Distillation
MAGA distills specialized GUI agents into one policy by allocating supervision according to structured-action correctness rather than treating all response tokens equally. A training-only teacher hint supports fusion across mobile, web and desktop settings without changing student inputs.
Cross-platform
Tactile: Giving Computer-Using Agents Hands and Feet
Tactile turns accessibility, OCR and visual evidence into explicit desktop targets with executable actions and verification cues. Its execution loop chooses semantic actions or coordinate fallbacks while retaining provenance for replay and diagnosis.
DesktopmacOS
LUMOS: A Semantic Operating-System Layer for Accessibility-Grounded AI Agents
LUMOS converts native accessibility metadata and browser UI structure into semantic interface blueprints with stable element identifiers and available actions. It combines live pointer-to-element grounding with an observe-act loop constrained to visible-UI primitives.
Web
TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents
TUA-Bench evaluates terminal-use agents on general digital tasks, including document editing, email, web information seeking and specialist software workflows. Tasks run in real terminals with deterministic setup and execution-based scoring.
Adjacent research
Beyond the GUI Paradigm: Do Mobile Agents Need the Phone Screen?
This study compares general coding agents using mobile command-line access with reproducible GUI agents on AndroidWorld and MobileWorld. It also supplies oracle CLI analyses and a separate task suite for operations such as bulk changes, filtering and hidden device state.
AndroidAdjacent research
PreAct: Computer-Using Agents that Get Faster on Repeated Tasks
PreAct compiles successful screen-interaction runs into guarded state-machine programs for repeated tasks. It validates programs from clean states before storage and returns control to the agent when replayed screen expectations fail, with tests across mobile, desktop and web.
Cross-platform
ComAct: Reframing Professional Software Manipulation via COM-as-Action Paradigm
ComAct proposes COM program synthesis as an alternative interface for manipulating professional software and introduces ComCADBench with GUI-based comparison agents. ComActor adds a staged training and correction approach, supported by a Windows-container training platform.
DesktopWindowsAdjacent research
CLI-Anything: Towards Agent-Native Computer Use
CLI-Anything argues for application interfaces built around structured commands, explicit state and execution feedback. It describes transforming existing software into command-line harnesses and organizing that approach through CLI-Hub.
Adjacent research
Syll: Open-Source Personal Automation with Cross-Surface Execution
Syll combines tool calls, shell execution and visual GUI control in a personal-automation runtime. It turns user demonstrations into reusable skills and exposes execution logs, keyframes and approval checkpoints for inspection.
DesktopmacOS
GUI Agents for Continual Game Generation
PlaytestArena evaluates generated browser games using expected in-play behaviors adjudicated by GUI agents. Play2Code pairs a coding agent with a rubric-blind GUI playtester whose interaction traces and feedback guide iterative refinement, with a separate final judge.
WebAdjacent research
VISTA: An End-to-End Benchmark for Visual Spec-to-Web-App Coding Agents
VISTA evaluates coding agents that turn design handoffs into working web and Android applications. Its evaluator locates annotated interactive components and probes their behavior, keeping visual localization distinct from functional correctness.
WebAndroidAdjacent research
TClone: Low-Latency Forking of Live GUI Environments for Computer-Use Agents
TClone versions live GUI workspaces through snapshots, isolated forks, rollback and selective commit or merge. It separates fast branch creation from durable checkpoints using copy-on-write sharing, filesystem versioning and asynchronous checkpointing, with end-to-end agent-loop latency measurements.
MMTB: Evaluating Terminal Agents on Multimedia-File Tasks
MultiMedia-TerminalBench evaluates agents that use terminal tools to work with audio and video files. The Terminus-MM harness adds multimedia perception to study how accessible evidence changes executable workflows and task outcomes.
Adjacent research
Training Computer Use Agents to Assess the Usability of Graphical User Interfaces
uxCUA learns to assess interface usability by choosing interaction flows, executing them and predicting a numerical usability score. Training uses interactive UIs with usability labels and human preferences, with evaluations on synthetic and real interfaces.
PlayCoder: Making LLM-Generated GUI Code Playable
PlayEval measures whether generated GUI applications remain logically playable, using PlayTester to execute task-oriented GUI playthroughs. PlayCoder closes the loop between repository-aware generation, interactive testing and targeted code repair.
WebXSkill: Skill Learning for Autonomous Web Agents
WebXSkill pairs parameterized action programs with step-level language guidance, mining reusable skills from synthetic browser-agent traces. A URL-based graph supports retrieval, and deployment alternates between direct program execution and guided native planning.
Web
GPA: Learning GUI Process Automation from Demonstrations
Presents demonstration-based GUI process replay with Sequential Monte Carlo localization and readiness checks. GPA executes the learned workflow locally and can serve as the GUI execution component of an agent using MCP or CLI tools.
WebTestBench: Evaluating Computer-Use Agents towards End-to-End Automated Web Testing
Introduces WebTestBench for end-to-end website testing, separating checklist generation from defect detection. Its WebTester baseline evaluates test coverage, detection and sustained interaction across web application categories.
Web
CarePilot: A Multi-Agent Framework for Long-Horizon Computer Task Automation in Healthcare
CareFlow benchmarks long healthcare-software workflows, while CarePilot coordinates action proposals, execution checks and corrective feedback using short- and long-term memory. Its actor consumes visual interfaces and system state to select semantic actions.
SpecOps: A Fully Automated AI Agent Testing Framework in Real-World GUI Environments
SpecOps divides testing of GUI-based agents into test generation, environment setup, execution and validation handled by specialist agents. Its evaluation spans five real-world agents, including web applications and browser extensions, and measures execution reliability and bug discovery.
Web
Avenir-UX: Automated UX Evaluation via Simulated Human Web Interaction with GUI Grounding
Avenir-UX combines grounded web interaction with simulated user profiles and structured usability measures to produce evaluations of website journeys. The paper describes its architecture and use of SUS, step-wise ease questions and think-aloud traces for automated UX reporting.
Web
EmbeWebAgent: Embedding Web Agents into Any Customized UI
EmbeWebAgent embeds agent access into existing web applications through frontend observations and a page-specific function registry. Its workflow combines GUI primitives, composite actions and MCP tools, demonstrated in a live interface.
Web
WebTestPilot: Agentic End-to-End Web Testing against Natural Language Specification by Inferring Oracles with Symbolized GUI Elements
WebTestPilot turns natural-language requirements into web test steps with inferred preconditions and postconditions over symbolized GUI elements. A benchmark of bug-injected web applications tests whether these context-aware oracles distinguish application defects from agent mistakes.
Web
See, Plan, Snap: Evaluating Multimodal GUI Agents in Scratch
ScratchWorld evaluates program construction through the Scratch interface using fine-grained drag-and-drop actions and a complementary semantic-API mode. Runtime tests assess the resulting programs, allowing analysis of the gap between program reasoning and GUI manipulation.
Web
CovAgent: Overcoming the 30% Curse of Mobile Application Coverage with Agentic AI and Dynamic Instrumentation
CovAgent augments Android GUI fuzzers by analyzing decompiled app code and component transitions to identify unsatisfied activation conditions. A second agent generates instrumentation that enables otherwise difficult-to-reach activities, with coverage-based evaluation.
Android
CUA-Skill: Develop Skills for Computer Using Agent
CUA-Skill encodes Windows application knowledge in parameterized skills and execution/composition graphs. Its agent retrieves and instantiates these skills and uses memory for failure recovery, with end-to-end evaluation on WindowsAgentArena.
DesktopWindows
GUITester: Enabling GUI Agents for Exploratory Defect Discovery
Introduces GUITestBench for interactive defect discovery and GUITester, which separates task navigation from anomaly verification. Testing-oriented plans and history-based reflection help distinguish software defects from agent execution mistakes.
GAIR: GUI Automation via Information-Joint Reasoning and Group Reflection
Introduces GAIR, which combines information from specialized GUI models through a general-purpose coordinating model. When the available information is insufficient, group reflection sends targeted requests back to the specialists before selecting an operation.
In-Browser LLM-Guided Fuzzing for Real-Time Prompt Injection Testing in Agentic AI Browsers
Proposes an in-browser fuzzing framework that uses language-model guidance to test indirect prompt-injection vulnerabilities in agentic browsers. The work focuses on unwanted actions triggered by webpage instructions under the browser agent's user privileges.
Web
From Imperative to Declarative: Towards LLM-friendly OS Interfaces for Boosted Computer-Use Agents
Introduces a Declarative Model Interface that maps existing GUIs to access, state and observation primitives while handling low-level interactions. Evaluation uses Word, PowerPoint and Excel on Windows with a GUI-agent baseline.
DesktopWindows
WALT: Web Agents that Learn Tools
WALT explores website functionality and converts demonstrated interactions into validated, reusable tools combining DOM actions, URL navigation, and agentic fallbacks. It evaluates this expanded browser action space on WebArena and VisualWebArena.
Web
ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data
Introduces a cross-platform computer-use dataset and agents trained with a collection pipeline combining automated interaction and human input.
Cross-platform
You Don't Know Until You Click:Automated GUI Testing for Production-Ready Software Evaluation
RealDevWorld evaluates generated software repositories with 194 development tasks and AppEvalPilot, an agent judge that operates the resulting GUIs. The evaluation examines functional behavior, visual fidelity, and runtime responses to produce task-specific diagnostic feedback.
CoAct-1: Computer-using Multi-Agent System with Coding Actions
CoAct-1 delegates subtasks between a GUI operator and a programmer that executes Python or Bash. Its OSWorld evaluation studies a hybrid action space in which scripts handle suitable file and data operations while visual interaction handles other steps.
Desktop
OS-MAP: How Far Can Computer-Using Agents Go in Breadth and Depth?
OS-MAP organizes 416 computer-use tasks across 15 applications by a five-level automation taxonomy and a hierarchy of user demands. The benchmark separates required autonomy from generalization scope to diagnose where agents struggle as task demands increase.
Scenario-Guided LLM-based Mobile App GUI Testing
ScenGen organizes mobile GUI testing around business scenarios rather than unrestricted exploration. Five collaborating agents observe the interface, choose and execute actions, check scenario completion, and record operations and runtime bugs.
LiteCUA: Computer as MCP Server for Computer-Use Agent on AIOS
AIOS 1.0 exposes computer states and actions through an MCP server to provide structured environmental context for agents. LiteCUA is a lightweight agent built on this interface, evaluated on OSWorld to study the effect of computer-environment contextualization.
Desktop
UFO2: The Desktop AgentOS
Introduces UFO2, a Windows desktop agent architecture combining a coordinating HostAgent, application-specialized agents, and a shared GUI/API action layer. It integrates UI Automation, visual parsing, speculative planning, and an isolated picture-in-picture desktop.
DesktopWindows
VLM-Fuzz: Vision Language Model Assisted Recursive Depth-first Search Exploration for Effective UI Testing of Android Apps
VLM-Fuzz combines depth-first UI exploration with Android manifest analysis, runtime widget hierarchies, and on-demand visual reasoning to generate app test inputs. It evaluates coverage on 59 benchmark apps and uses additional Google Play apps to study crash discovery.
Android
ReuseDroid: A VLM-empowered Android UI Test Migrator Boosted by Active Feedback
ReuseDroid migrates Android GUI tests between functionally similar apps using VLM agents that separate shared test intent from app-specific interaction flows. It introduces LinPro, a dataset of 578 migration tasks across 39 apps, to evaluate test transfer.
Android
Are Autonomous Web Agents Good Testers?
Adapts web agents to execute natural-language test steps, verify assertions, and issue test verdicts. The study introduces 113 passing and failing test cases across three offline web applications and compares two agent implementations with qualitative failure analysis.
Web
API Agents vs. GUI Agents: Divergence and Convergence
Compares API-driven and GUI-driven agents across architecture, development, and interaction considerations, discussing when hybrid approaches are useful.
Cross-platform
Towards Enterprise-Ready Computer Using Generalist Agent
Describes the CUGA enterprise-agent development process, emphasizing iterative evaluation, failure analysis, and refinement on software-task benchmarks.
Web
Programming with Pixels: Can Computer-Use Agents do Software Engineering?
Introduces a visual IDE environment and benchmark for software-engineering tasks, comparing screenshot-based control with access to editing and shell tools.
Desktop
GUI Testing Arena: A Unified Benchmark for Advancing Autonomous GUI Testing Agent
Evaluates GUI testing through test-intent generation, execution, and defect detection using real, modified, and synthetic mobile application data.
Large Action Models: From Inception to Implementation
Uses a Windows-agent case study to describe a development workflow for large action models, covering data collection, training, grounding, environment integration, and evaluation.
DesktopWindows
The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Use
Presents a preliminary study of Claude 3.5 Computer Use across desktop tasks and software, analyzing planning, action, and critique behavior and providing a framework for deploying GUI-automation models.
Desktop
AutoGLM: Autonomous Foundation Agents for GUIs
Presents AutoGLM agents for browser and phone control, separating planning from grounding and training through an evolving interaction curriculum.
WebAndroid
AgentStore: Scalable Integration of Heterogeneous Agents As Specialized Generalist Computer Assistant
Introduces a platform that integrates specialized agents under a coordinating MetaAgent for varied computer tasks.
Desktop
Showing 1–60 of 111 resources.
Get full records and evidence through the topic export manifest. Source records can be passed to the standard RAG and citation exporters after cloning the catalog.