Datasets · curated
AgentNet
22.6K human-annotated desktop computer-use tasks on Windows, macOS and Ubuntu, released with OpenCUA for training and evaluating vision-language-action models.
Release date not recorded · desktopRESOURCE CATEGORY
Computer-use agent datasets
Browse 86 reviewed datasets connected to computer-use, GUI, web, and mobile agents. Each entry links to primary evidence and records its scope.
Review means AI-assisted editorial inspection, not independent reproduction. Unknown licenses and specifications remain unknown.
Compare source-reviewed specifications
Datasets · curated
AgentSynth tasks and trajectories
Published AgentSynth task definitions and recorded GUI trajectories, with OSWorld and InSTA training configurations and packaged screenshots, thoughts and commands.
Release date not recorded · desktop, webDatasets · curated
AgentTrek
Synthesizes browser-agent trajectories by replaying web tutorials as guidance, releasing the trajectory data with multimodal grounding and reasoning and an AgentTrek-32B agent.
Release date not recorded · webDatasets · curated
Aguvis stage 1 grounding data
The first-stage Aguvis training collection, packaging GUI element-grounding annotations and image archives from multiple interface-data sources.
Release date not recorded · cross-platformDatasets · curated
Aguvis stage 2 trajectory data
The second-stage Aguvis collection of interface trajectories for learning action sequences, planning and reasoning after grounding pretraining.
Release date not recorded · cross-platformDatasets · curated
AMEX
Code for AMEX Android GUI data that converts its three annotation levels into question-answer pairs and fine-tunes a LLaMA2-Accessory model, with a released checkpoint.
Release date not recorded · androidDatasets · curated
Android in the Wild
Human demonstrations of Android tasks with screenshots and touch actions.
Release date not recorded · androidDatasets · curated
AndroidCode
HTML and annotation release used for Code2World’s action-conditioned GUI-state prediction training, constructed from mobile GUI trajectories.
Release date not recorded · androidDatasets · curated
AndroidControl
Android demonstrations with high-level goals, step instructions, screenshots, and accessibility trees.
Release date not recorded · androidDatasets · curated
Aria-UI Context-aware Data
Episode-based mobile GUI grounding data released with Aria-UI for incorporating interaction history into target localization.
Release date not recorded · androidDatasets · curated
Aria-UI Data
GUI element descriptions and grounding instructions spanning web, mobile, and Ubuntu desktop interfaces.
Release date not recorded · web, android, desktop, linuxDatasets · curated
AutoDroid V2 training data
Conversation-formatted supervision released for fine-tuning AutoDroid V2, with the project documenting a ShareGPT-style JSON configuration.
Release date not recorded · androidDatasets · curated
AutoGUI v1 training data
Screenshot-based grounding and captioning samples with UI functionality descriptions, target coordinates, and bounding boxes.
Release date not recorded · web, cross-platformDatasets · curated
CAGUI
Chinese Android GUI benchmark released with AgentCPM-GUI, with a grounding track for widgets and OCR text and an agent track for multi-step tasks in Chinese apps.
Release date not recorded · androidDatasets · curated
CMGUI
18k trajectories with 98k steps from more than 50 Chinese mobile apps in domains such as e-commerce and social media, for building GUI agents on Chinese apps.
Release date not recorded · androidDatasets · curated
CUA-Gym
Computer-use environment and task resources for agent research.
Release date not recorded · desktopDatasets · curated
DragOn
Drag-grounding training and evaluation data pairing each screenshot instruction with start and end boxes, across text highlighting, spreadsheet cells, slide resizing and sliders.
Release date not recorded · cross-platformDatasets · curated
Dreamer V1 Data
Web-world-model training data with prompts, images, action descriptions, and responses, released with the WebDreamer project.
Release date not recorded · webDatasets · curated
DroidCall data
Instruction-to-function-call examples for learning Android intent invocation, with training, testing and API-annotation files for mobile task execution.
Release date not recorded · androidDatasets · curated
FingerTip-20K
Over 20,000 human Android episodes across 500+ apps with user context and long-term usage patterns, plus tracks for proactive intent prediction and personalized execution.
Release date not recorded · androidDatasets · curated
Go-Browse-WA processed data
Flattened web-interaction training records with accessibility-tree observations, assistant completions, trajectory rewards and step-link metadata.
Release date not recorded · webDatasets · curated
GroundGUI RL Dataset
Desktop grounding prompts selected from GroundCUA and formatted for click-based reinforcement learning with target-box rewards.
Release date not recorded · desktopDatasets · curated
Grounding-Anything-GUI
The public training archive linked by the Phi-Ground project for its Phi-Ground-Any model.
Release date not recorded · unspecifiedDatasets · curated
GUI-360
Desktop GUI trajectory data supporting multiple interface understanding tasks.
Release date not recorded · desktop, windowsDatasets · curated
GUI-Action Narrator
Action narration resources for GUI interaction trajectories.
Release date not recorded · cross-platformDatasets · curated
GUI-Actor data
A processed screenshot and bounding-box collection released for training GUI-Actor visual grounding models.
Release date not recorded · unspecifiedDatasets · curated
GUI-CIDER causal knowledge data
A published JSONL corpus of synthesized planning and causal knowledge for GUI-agent mid-training, linked directly from the official GUI-CIDER project.
Release date not recorded · androidDatasets · curated
GUI-Drag
Data for learning text-dragging interactions in graphical interfaces.
Release date not recorded · cross-platformDatasets · curated
GUI-Lasagne L1
The published GUI-Lasagne training-data collection used for SpiritSight, distributed as dated archive shards for visual interface learning.
Release date not recorded · unspecifiedDatasets · curated
GUI-Net-1M
A TongUI-collected training corpus from multimodal web tutorials, distributed as annotation files and split screenshot archives for GUI trajectory supervision.
Release date not recorded · unspecifiedDatasets · curated
GUI-Odyssey
Human-demonstrated Android episodes for cross-app navigation, spanning six task types, 212 apps and about 1,400 app combinations, for training and evaluating agents.
Release date not recorded · androidDatasets · curated
GUI-R1 data bundle
Training and evaluation Parquet bundles accompanying GUI-R1, including GUI grounding and mobile/web action-prediction data.
Release date not recorded · desktop, windows, linux, macos, android, webDatasets · curated
GUI-World
GUI video benchmark and training data covering six scenarios, from Android and iOS to websites and XR, with eight question types for image and video LLMs and the GUI-Vid model.
Release date not recorded · cross-platformDatasets · curated
GUIAct
Web and Android GUI action-instruction data, with single-step, multi-step and smartphone partitions.
Release date not recorded · web, androidDatasets · curated
GUIChat
Conversational GUI supervision with text-rich images and bounding boxes, organized as single-turn questions and multi-turn dialogues.
Release date not recorded · unspecifiedDatasets · curated
GUICourse
Three datasets for turning general VLMs into GUI agents: GUIEnv for OCR and grounding, GUIAct with 67k single-step and 15k multi-step web and Android actions, and GUIChat dialogues.
Release date not recorded · cross-platformDatasets · curated
GUIEnv-local
The released local-supervision portion of GUIEnv, pairing webpage regions and text for OCR and GUI grounding.
Release date not recorded · webDatasets · curated
GUIMid training data
A released GUI-agent mid-training mixture and supporting domain datasets, together with a post-training trajectory file and image archives for web and mobile interaction.
Release date not recorded · web, androidDatasets · curated
GUIOdyssey — current data release
Cross-app Android interaction episodes with fine-grained step instructions, screen descriptions, action context, and target boxes.
Release date not recorded · androidDatasets · curated
Jedi
Synthetic GUI grounding data associated with UI decomposition research and the OSWorld-G benchmark.
Release date not recorded · desktopDatasets · curated
Mobile3M
Pre-training corpus for Xiaomi's MobileVLM, distributed as per-app graph archives for mobile apps such as Baidu Map, QQ Music, Ctrip and Vipshop.
Release date not recorded · androidDatasets · curated
MobileViews
Over 600,000 Android screenshot and view-hierarchy pairs from more than 20,000 Google Play apps, collected with an optimized DroidBot, with complete per-app traces.
Release date not recorded · androidDatasets · curated
MONDAY
Cross-platform mobile GUI trajectory dataset derived from instructional videos, with released data-processing helpers and example evaluation code for Android and iOS data.
Release date not recorded · android, ios, cross-platformDatasets · curated
MONDAY annotations
Video-derived mobile navigation annotations that pair scene boundaries and screen or element boxes with action labels for training and evaluating GUI agents.
Release date not recorded · android, iosDatasets · curated
MoTIF
Mobile task demonstrations and natural-language instructions.
Release date not recorded · androidDatasets · curated
MT-Mind2Web
Multi-turn web-navigation conversations constructed from Mind2Web interactions, with dialogue turns, HTML snapshots, element candidates and click/type/select action sequences.
Release date not recorded · webDatasets · curated
MultiUI
7.3 million training samples built from webpage UIs and structured with enhanced accessibility trees, for text-rich visual understanding, released with UIX-Qwen2 models.
Release date not recorded · webDatasets · curated
NNetNav Live data
Browser-interaction data released with NNetNav for its live-website setting, with separate training and test files.
Release date not recorded · webDatasets · curated
NNetNav WebArena data
NNetNav's WebArena-oriented browser-interaction data release, with separate training and test files.
Release date not recorded · webDatasets · curated
OpenMobile-Data-v2
Mobile-agent supervision from emulator and simulator trajectories, plus executable tasks with starting states and success checks for reinforcement learning.
Release date not recorded · android, webDatasets · curated
OS-Atlas Data
GUI grounding pre-training data for OS-Atlas in desktop (Linux, macOS, Windows), mobile and web domains, with element boxes stored as relative left, top, right and bottom ratios.
Release date not recorded · cross-platformDatasets · curated
OS-Genesis
Interaction-driven pipeline that synthesizes mobile and web GUI agent trajectories without human supervision or predefined tasks, via reverse task synthesis and a trajectory reward model.
Release date not recorded · cross-platformDatasets · curated
OS-Genesis mobile training data
Reverse-task-synthesis mobile trajectories released in separate AndroidControl and AndroidWorld training annotation files.
Release date not recorded · androidDatasets · curated
PC-Agent-E training data
Released dataset used to train PC-Agent-E, linked by the official pipeline for refining human trajectories and synthesizing action reasoning.
Release date not recorded · desktop, windows, linuxDatasets · curated
Plan-and-Act training data
Planner and action-executor training examples for WebArena-Lite and WebVoyager, with model-specific processed files and raw trajectory annotations.
Release date not recorded · webDatasets · curated
Rico
Mobile app screenshots, view hierarchies, and interaction traces for UI research.
Release date not recorded · androidDatasets · curated
RoTS
Announced GUI trajectory synthesis resources for recovery from policy-induced errors; the inspected repository marks code and data releases as in preparation.
Release date not recorded · cross-platformDatasets · curated
ScaleCUA Data
Cross-platform training data released with ScaleCUA, with grounding and navigation annotations for Ubuntu, Windows, macOS, Android, iPhone and web screenshots.
Release date not recorded · cross-platformDatasets · curated
ScienceBoard evaluation trajectories
Archives of ScienceBoard agent evaluation runs, separated by agent configuration and screenshot, accessibility-tree or set-of-marks observation mode.
Release date not recorded · desktop, linuxDatasets · curated
Screen2Words
Mobile screen descriptions for language-based screen understanding.
Release date not recorded · androidDatasets · curated
ScreenDrag
GUI dragging data with screen observations, prior state, and continuous pointer actions, provided as trajectory files and Parquet representations.
Release date not recorded · desktopDatasets · curated
ScreenQA
About 86,000 human-written question-answer pairs over 35,000 Rico mobile screenshots, plus ScreenQA Short and ComplexQA variants, for benchmarking screen content understanding.
Release date not recorded · androidDatasets · curated
ScreenRef
GUI screenshots paired with localization instructions and targets, including absent-target examples; image access is gated under publisher terms.
Release date not recorded · desktop, web, android, linuxDatasets · curated
ScreenRef Annotations
Annotation files, task recipes, and image manifests for ScreenRef; screenshots reside in the separately gated image repository.
Release date not recorded · desktop, web, android, linuxDatasets · curated
ScreenSpot-Pro data
High-resolution professional interface screenshots and target annotations for evaluating instruction-based GUI localization.
Release date not recorded · desktop, windows, macos, linuxDatasets · curated
ShowUI Desktop
7,496 desktop grounding examples built from OmniAct screenshots, with annotations rewritten by GPT-4o to describe element appearance, spatial relations and function.
Release date not recorded · desktopDatasets · curated
ShowUI-web
Web-interface grounding examples with instructions, element boxes and points, paired with a separately packaged screenshot archive.
Release date not recorded · webDatasets · curated
ShowUI-π training data
Drag-and-click episode data for the ShowUI-π model, spanning desktop editing, presentation, handwriting and other interface tasks.
Release date not recorded · desktopDatasets · curated
UGround V1 Data
Gated web grounding training data for UGround in Web-Hybrid and Web-Direct subsets, with a bounding-box version whose coordinates are normalized to 0-999.
Release date not recorded · cross-platformDatasets · curated
UI-AGILE-Data
UI-AGILE training annotations and screenshots together with converted GUI-grounding and AndroidControl evaluation files.
Release date not recorded · androidDatasets · curated
UI-Genie reward data
Mobile GUI reward annotations released for training UI-Genie’s trajectory-assessment model.
Release date not recorded · androidDatasets · curated
UI-Genie trajectory data
Synthetic mobile GUI trajectory supervision and screenshots released with the UI-Genie agent-training pipeline.
Release date not recorded · androidDatasets · curated
UI-Ins
Treats grounding instructions as multiple reasoning perspectives, releasing a data pipeline, training code and UI-Ins-7B and 32B models trained by SFT on synthesized instructions, then RL.
Release date not recorded · cross-platformDatasets · curated
UI-R1 3B training data
The released mobile training subset for the original UI-R1 3B model, with action annotations and individual screenshot files.
Release date not recorded · androidDatasets · curated
UI-S1 released training examples
A public example subset of the UI-S1 mobile training data, packaged as JSONL annotations and Parquet images.
Release date not recorded · androidDatasets · curated
UI-Vision
Desktop GUI data for interface understanding and interaction research.
Release date not recorded · desktopDatasets · curated
UIBert AppSim and RefExp annotations
Two Rico-derived UI component retrieval datasets: AppSim pairs semantically similar elements across screens, while RefExp links referring expressions to target elements.
Release date not recorded · androidDatasets · curated
UICrit
Mobile-interface critiques and quality ratings with region coordinates and RICO screen references, including a larger three-annotator release than the original paper dataset.
Release date not recorded · unspecifiedDatasets · curated
VideoCAD data
CAD screen recordings paired with action logs and target model images for studying precise interface control and long-horizon visual imitation.
Release date not recorded · webDatasets · curated
Wave UI
Screenshot-instruction records with target boxes and UI metadata such as OCR, source, element type and platform.
Release date not recorded · unspecifiedDatasets · curated
WebEvolver SFT data
Two JSONL collections for supervised fine-tuning the WebEvolver agent policy and web world model.
Release date not recorded · webDatasets · curated
WebPRM checklist-generation data
Web-task context, starting observations and action history paired with structured checklists and subgoal analyses for checklist generation.
Release date not recorded · webDatasets · curated
WebPRM Collection preference data
Checklist-annotated web decision records with page observations, prior actions, one preferred continuation and rejected alternatives for process-reward learning.
Release date not recorded · webDatasets · curated
WebSRC v1.0
Webpage segments, questions, answers, HTML, screenshots and layout metadata for structural reading comprehension.
Release date not recorded · webDatasets · curated
WebUI
Web screenshot and interface data for UI research.
Release date not recorded · webDatasets · curated
WorkDemo-350
Desktop demonstrations with task descriptions, videos, input logs, and action-linked screenshot evidence for studying reusable agent skills.
Release date not recorded · desktop