Where to find training data for GUI agents
Start with the supervision your model needs: screen understanding, element grounding, individual actions, or complete interaction trajectories. Then match the platform, annotation format, split, and license before downloading.
Which data types are useful?
- Screen understanding: screenshots paired with descriptions, questions, or structural annotations.
- Grounding: an instruction and screen paired with a target element or location.
- Action prediction: observations and instructions paired with the next action.
- Trajectories: sequences of observations and actions that show how a task unfolds.
These supervision types support different objectives. More files or more screenshots do not automatically mean more complete task demonstrations.
What are concrete starting points?
| Resource | What to inspect |
|---|---|
| Android in the Wild | Android task demonstrations with screens, instructions, and actions; distinguish episodes from action examples. |
| AndroidControl | High-level task goals, step instructions, screenshots, accessibility trees, and action annotations. |
| AgentNet | Computer-use demonstrations associated with the OpenCUA research stack. |
| OS-Atlas data | GUI grounding data and the publisher’s access and licensing declaration. |
| Mind2Web | Web-task demonstrations and generalization splits across sites and domains. |
Compare source-reviewed dataset formats, counting units, and splits.
Can I use benchmark examples for training?
Check the intended split and evaluation protocol first. Training on examples used to measure generalization can invalidate the comparison you wanted to make. Preserve the split definitions and document any overlap or filtering decisions rather than treating every downloadable task as training data.
Does an open code repository imply open training data?
No. Code, datasets, and model weights may be distributed under different licenses. Check the current dataset card, original collection terms, access restrictions, and any upstream components. AwesomeCUA records these declarations separately and leaves unknown values empty; the catalog does not grant rights to the linked assets.
How do I compare dataset scale?
Keep units explicit: tasks, episodes, actions, screenshots, instructions, and UI elements are different quantities. A hosting platform’s size category may count packaged files rather than semantic examples. Compare the quantity that matches the intended training objective.
Continue with the reviewed dataset directory or searchable dataset catalog. Use our metadata export to inspect sources and licenses programmatically; it contains resource records, not the underlying training data.
Editorial update: 2026-10-16. Dataset suitability and permission depend on the original asset and intended use.