Data generation

On the Effects of Data Scale on UI Control Agents

Wei Li, William Bishop et al.

Studies how Android task performance changes with demonstration scale and releases AndroidControl with high- and low-level instructions.

Large Language Models Can Self-Improve At Web Agent Tasks

Ajay Patel, Markus Hofmarcher et al.

Examines self-improvement by fine-tuning web agents on three mixtures of model-generated training data. WebArena experiments and trajectory-level evaluation measures study task performance, robustness, and behavior beyond aggregate success rates.

MUD: Towards a Large-Scale and Noise-Filtered UI Dataset for Modern Style UI Modeling

Sidong Feng, Suyu Ma et al.

MUD combines LLM-guided Android app exploration, noise filtering and human annotation to collect modern UI screenshots. The study evaluates the resulting data for element detection and UI retrieval.

GUIDE: Graphical User Interface Data for Execution

Rajat Chawla, Adarsh Jha et al.

Introduces annotated website task trajectories with screenshots, prior actions, reasoning, and grounded next actions for process automation.

WILBUR: Adaptive In-Context Learning for Robust and Accurate Web Agents

Michael Lutz, Arth Bohra et al.

WILBUR learns to rank prior task demonstrations for a web agent’s prompt and synthesizes instructions from past runs. A generated task curriculum and a backtracking mechanism support adaptation and recovery, evaluated on WebVoyager.

AutoWebGLM: A Large Language Model-based Web Navigating Agent

Hanyu Lai, Xiao Liu et al.

Builds a web agent through simplified HTML observations, human-and-AI browsing data, curriculum training, and reinforcement-learning refinement.

Computer User Interface Understanding. A New Dataset and a Learning Framework

Andrés Muñoz, Daniel Borrajo

Introduces desktop interaction videos and a framework combining synthetic examples with hierarchical contrastive learning for interface classification.

WebLINX: Real-World Website Navigation with Multi-Turn Dialogue

Xing Han Lù, Zdeněk Kasner et al.

Collects expert demonstrations of conversational web navigation and studies retrieval-based element selection for processing large pages.

ScreenAI: A Vision-Language Model for UI and Infographics Understanding

Gilles Baechler, Srinivas Sunkara et al.

Introduces ScreenAI and screen-annotation training that supports UI question answering, navigation, and summarization data generation.

SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents

Kanzhi Cheng, Qiushi Sun et al.

Trains a screenshot-driven GUI agent with grounding pretraining and introduces ScreenSpot for evaluating interface target localization.

WebVLN: Vision-and-Language Navigation on Websites

Qi Chen, Dileepa Pitawela et al.

WebVLN trains website navigation from question-based instructions using both rendered visual information and underlying HTML. It introduces WebVLN-v1 and a website-aware navigation network to study this digital variant of vision-and-language navigation.

ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations

Yue Jiang, Eldon Schoop et al.

ILuvUI generates 335,000 conversational training examples paired with UI screenshots, covering questions, descriptions, and plans. The resulting vision-language model is assessed on element detection, response quality, and examples of multi-step UI navigation.

Android in the Wild: A Large-Scale Dataset for Android Device Control

Christopher Rawles, Alice Li et al.

Collects Android interaction demonstrations with screens, actions, and instructions across device and platform variations for robustness evaluation.

Mind2Web: Towards a Generalist Agent for the Web

Xiang Deng, Yu Gu et al.

Introduces a web-task dataset spanning diverse websites and evaluates generalization to unseen sites and domains.

WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semantics

Jason Wu, Siyan Wang et al.

Collects rendered webpages and structural metadata for visual UI modeling, studying how web semantics transfer to interface-understanding tasks.

Lexi: Self-Supervised Learning of the UI Language

Pratyay Banerjee, Shweti Mahajan et al.

Introduces Lexi and UICaption to learn UI image-text representations from descriptions of interface functionality. Evaluation covers UI action entailment, instruction-based retrieval, referring-expression grounding, and UI entity recognition.

Understanding HTML with Large Language Models

Izzeddin Gur, Ofir Nachum et al.

Studies language models for HTML element classification, page description and autonomous web navigation. Fine-tuned models are evaluated on MiniWoB, and a distilled, automatically labeled CommonCrawl HTML corpus supports the training approach.

Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding

Kenton Lee, Mandar Joshi et al.

Pix2Struct pretrains an image-to-text model by reconstructing simplified HTML from masked webpage screenshots. Its variable-resolution visual representation is evaluated across several domains, including user-interface understanding.

Towards Better Semantic Understanding of Mobile Interfaces

Srinivas Sunkara, Maria Wang et al.

Augments RICO screens and view hierarchies with human labels for icon shape and meaning and for links between UI elements and text labels. It compares image-only and multimodal models on semantic understanding of previously unseen apps.

Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scale

Gang Li, Gilles Baechler et al.

Introduces CLAY, a pipeline that repairs noisy mobile UI layouts by removing nodes without valid visual counterparts and assigning semantic element types, supported by human-annotated layouts for evaluation.

Understanding Mobile GUI: from Pixel-Words to Screen-Sentences

Jingwen Fu, Xiaoyi Zhang et al.

Models mobile screens as atomic text and graphic components whose relationships are learned by a Screen Transformer. The work introduces RICO-PW annotations and evaluates metadata-free screen understanding on relation, clickability, retrieval, and app-classification tasks.

Mapping Natural Language Instructions to Mobile UI Action Sequences

Yang Li, Jiacong He et al.

Introduces PIXELHELP for predicting mobile UI action sequences from instructions, together with action-phrase annotations and synthetic UI action descriptions for training. Separate Transformers extract instruction steps and ground object descriptions using interface content and screen position.

Mapping Natural Language Commands to Web Elements

Panupong Pasupat, Tian-Shun Jiang et al.

Defines language grounding on webpages as choosing the UI element targeted by a natural-language command. A collection of more than 50,000 commands and three baseline models study functional references, relationships between elements, and visual position cues.

Rico: A Mobile App Dataset for Building Data-Driven Design Applications

Biplab Deka, Zifeng Huang et al.

Introduces Rico, collecting Android interface designs and interaction traces through crowdsourced use followed by automated exploration. The paper demonstrates UI-layout retrieval using learned representations and describes visual, textual, structural, and interaction data.

World of Bits: An Open-Domain Platform for Web-Based Agents

Tianlin Shi, Andrej Karpathy et al.

Introduces World of Bits, a web-task platform combining human demonstrations, keyboard and mouse actions, and cached HTTP traffic for repeatable task environments. It evaluates agents trained through behavioral cloning and reinforcement learning.

AgentNet

22.6K human-annotated desktop computer-use tasks on Windows, macOS and Ubuntu, released with OpenCUA for training and evaluating vision-language-action models.

AgentSynth tasks and trajectories

Published AgentSynth task definitions and recorded GUI trajectories, with OSWorld and InSTA training configurations and packaged screenshots, thoughts and commands.

AgentTrek

Synthesizes browser-agent trajectories by replaying web tutorials as guidance, releasing the trajectory data with multimodal grounding and reasoning and an AgentTrek-32B agent.

Aguvis stage 1 grounding data

The first-stage Aguvis training collection, packaging GUI element-grounding annotations and image archives from multiple interface-data sources.

Aguvis stage 2 trajectory data

The second-stage Aguvis collection of interface trajectories for learning action sequences, planning and reasoning after grounding pretraining.

AMEX

Code for AMEX Android GUI data that converts its three annotation levels into question-answer pairs and fine-tunes a LLaMA2-Accessory model, with a released checkpoint.

Android in the Wild

Human demonstrations of Android tasks with screenshots and touch actions.

AndroidCode

HTML and annotation release used for Code2World’s action-conditioned GUI-state prediction training, constructed from mobile GUI trajectories.

AndroidControl

Android demonstrations with high-level goals, step instructions, screenshots, and accessibility trees.

AutoDroid V2 training data

Conversation-formatted supervision released for fine-tuning AutoDroid V2, with the project documenting a ShareGPT-style JSON configuration.

CAGUI

Chinese Android GUI benchmark released with AgentCPM-GUI, with a grounding track for widgets and OCR text and an agent track for multi-step tasks in Chinese apps.

CMGUI

18k trajectories with 98k steps from more than 50 Chinese mobile apps in domains such as e-commerce and social media, for building GUI agents on Chinese apps.

CUA-Gym

Computer-use environment and task resources for agent research.

Dreamer V1 Data

Web-world-model training data with prompts, images, action descriptions, and responses, released with the WebDreamer project.

DroidCall data

Instruction-to-function-call examples for learning Android intent invocation, with training, testing and API-annotation files for mobile task execution.

FingerTip-20K

Over 20,000 human Android episodes across 500+ apps with user context and long-term usage patterns, plus tracks for proactive intent prediction and personalized execution.

GroundGUI RL Dataset

Desktop grounding prompts selected from GroundCUA and formatted for click-based reinforcement learning with target-box rewards.

Grounding-Anything-GUI

The public training archive linked by the Phi-Ground project for its Phi-Ground-Any model.

GUI-360

Desktop GUI trajectory data supporting multiple interface understanding tasks.

GUI-Action Narrator

Action narration resources for GUI interaction trajectories.

GUI-Actor data

A processed screenshot and bounding-box collection released for training GUI-Actor visual grounding models.

GUI-CIDER causal knowledge data

A published JSONL corpus of synthesized planning and causal knowledge for GUI-agent mid-training, linked directly from the official GUI-CIDER project.

GUI-Lasagne L1

The published GUI-Lasagne training-data collection used for SpiritSight, distributed as dated archive shards for visual interface learning.

GUI-Net-1M

A TongUI-collected training corpus from multimodal web tutorials, distributed as annotation files and split screenshot archives for GUI trajectory supervision.

GUI-Odyssey

Human-demonstrated Android episodes for cross-app navigation, spanning six task types, 212 apps and about 1,400 app combinations, for training and evaluating agents.

GUIAct

Web and Android GUI action-instruction data, with single-step, multi-step and smartphone partitions.

GUIChat

Conversational GUI supervision with text-rich images and bounding boxes, organized as single-turn questions and multi-turn dialogues.

GUICourse

Three datasets for turning general VLMs into GUI agents: GUIEnv for OCR and grounding, GUIAct with 67k single-step and 15k multi-step web and Android actions, and GUIChat dialogues.

GUIEnv-local

The released local-supervision portion of GUIEnv, pairing webpage regions and text for OCR and GUI grounding.

GUIOdyssey — current data release

Cross-app Android interaction episodes with fine-grained step instructions, screen descriptions, action context, and target boxes.

Jedi

Synthetic GUI grounding data associated with UI decomposition research and the OSWorld-G benchmark.

MONDAY

Cross-platform mobile GUI trajectory dataset derived from instructional videos, with released data-processing helpers and example evaluation code for Android and iOS data.

MONDAY annotations

Video-derived mobile navigation annotations that pair scene boundaries and screen or element boxes with action labels for training and evaluating GUI agents.

MultiUI

7.3 million training samples built from webpage UIs and structured with enhanced accessibility trees, for text-rich visual understanding, released with UIX-Qwen2 models.

NNetNav Live data

Browser-interaction data released with NNetNav for its live-website setting, with separate training and test files.