Recording

Work2Skill: Capturing Human GUI Experience for Reusable Agent Skill Induction

Yuqi Huang, Hui Li et al.

Aligns human screen recordings, input events, and accessibility information to derive reusable workflow skills from desktop demonstrations.

OmegaUse-SOP: SOP Engineering for Professional Computer Use from Human Demonstrations

Yixiong Xiao, Lang An et al.

OmegaUse-SOP converts expert GUI demonstrations into configurable procedures and refines them with domain rules and human input. It records, abstracts, grounds and verifies live execution, with a case study of photovoltaic simulation workflows in PVsyst 7.2.

Inducing Task Models from Computer-Use Traces

Yucheng Jiang, Zora Zhiruo Wang et al.

Task Model Induction disentangles concurrent activities in screenshot and mouse/keyboard traces, then constructs hierarchical goals and procedural control flow for each task. Evaluation tests reconstruction of observed execution and reuse of derived skills on held-out tasks.

LegacyWorld: Atomicity-Aware Evaluation of GUI Agents for Legacy Workflows

Thilo Reintjes, Sivajeet Chand et al.

LegacyWorld evaluates Windows GUI workflows under an atomicity criterion: finish correctly or fail without persistent unintended changes. It compares expert prompts with prompts derived from recorded golden paths and separates useful completion, safe failure and non-atomic side effects.

HANSEL: Extracting Breadcrumbs from Web Agent Trajectories for Interactive Verification

Yujin Zhang, Daye Nam

HANSEL extracts evidence pages and snippets from browser-agent traces and preserves relevant page state in interactive verification views. It flags untraceable answers and evaluates evidence selection and human verification effort separately.

PreAct: Computer-Using Agents that Get Faster on Repeated Tasks

Bojie Li

PreAct compiles successful screen-interaction runs into guarded state-machine programs for repeated tasks. It validates programs from clean states before storage and returns control to the agent when replayed screen expectations fail, with tests across mobile, desktop and web.

Syll: Open-Source Personal Automation with Cross-Surface Execution

Bo Zhang, Borui Zhang et al.

Syll combines tool calls, shell execution and visual GUI control in a personal-automation runtime. It turns user demonstrations into reusable skills and exposes execution logs, keyframes and approval checkpoints for inspection.

ShowUI-Aloha: Human-Taught GUI Agent

Yichun Zhang, Xiangwu Guo et al.

Transforms human screen recordings and input events into annotated demonstrations used by planning and execution components for GUI tasks.

GUIrilla: A Scalable Framework for Automated Desktop UI Exploration

Sofiya Garkot, Maksym Shamrai et al.

GUIrilla systematically explores macOS interfaces to collect interaction traces and accessibility metadata for downstream GUI research. It organizes these observations into hierarchical MacApp Trees and describes an accessibility-driven collection library.

Explorer: Robust Collection of Interactable GUI Elements

Iason Chaimalas, Arnas Vyšniauskas et al.

Explorer collects interactable GUI elements from a live target application and records user sessions for application-specific model training. It links repeated interface states into a navigation map that supports path planning and voice-directed interaction.

PC Agent: While You Sleep, AI Works -- A Cognitive Journey into Digital World

Yanheng He, Jiahe Jin et al.

Combines PC Tracker interaction capture, completion of action and reasoning annotations, and planning and grounding agents for desktop workflows.

Rousillon: Scraping Distributed Hierarchical Web Data

Sarah E. Chasins, Maria Mueller et al.

Introduces Rousillon, which generalizes a demonstration of collecting one hierarchical data row into a browser-automation program that traverses related pages. The paper develops relation-selection and generalization methods and evaluates users building such scripts.

SUGILITE: Creating Multimodal Smartphone Automation by Demonstration

Toby Jia-Jun Li, Amos Azaria et al.

Introduces SUGILITE, which combines spoken instructions, demonstrated Android actions, and UI hierarchies to generate reusable automation scripts. Its interface includes context checking and repairs for unfamiliar situations and changing app interfaces.

Ringer: Web Automation by Demonstration

Shaon Barman, Sarah Chasins et al.

Introduces Ringer, which converts demonstrated webpage interactions into scripts combining actions, target selectors, and waiting conditions. It evaluates replay robustness on tasks recorded from real websites.

Koala: Capture, Share, Automate, Personalize Business Processes on the Web

Greg Little, Tessa A. Lau et al.

Introduces Koala, a collaborative web automation system that records browser interactions as editable pseudo-natural-language scripts. Scripts are shared through a wiki and instantiated with personal or corporate data for reuse by other users.

WorkDemo-350

Desktop demonstrations with task descriptions, videos, input logs, and action-linked screenshot evidence for studying reusable agent skills.

Activity Recorder

Windows tray utility that records primary-display screenshots, keyboard key codes and mouse events as local JPEG and JSON files for training-data collection.

AgentStudio

Toolkit and Ubuntu-based environment for building and evaluating generalist computer-use agents through GUI and API interaction.

Browser Use WebUI

A Gradio interface for configuring and running Browser Use agents, including persistent browser sessions and interaction recordings.

BrowserOS / BrowserOS neo

Chromium-based browser implementations for external agent control and built-in assistant workflows, with session visibility and replay facilities.

OpenAdapt

Tools for learning and automating workflows from desktop demonstrations.

Scribe

Workflow recording and documentation product adjacent to agent data collection.

ShowUI-Aloha

Desktop agent that records demonstrations, derives semantic action traces, and uses them to plan and execute related workflows.

Work2Skill

Collects human GUI demonstrations and inspects multimodal evidence to derive reusable agent workflow skills.