OmniAct
Dataset and benchmark for generating computer interaction programs.
AI-assisted editorial review against a primary source. This is a catalog annotation, not an independent reproduction or endorsement.
What is this resource?
- Resource type
- Benchmarks
- Environment
- desktop, web
- Topics
- evaluation
- Record publication date
- Not recorded
- Evidence retrieved
- 2026-10-16T06:38:27Z
- Resource ID
- url:849a86beaee006c6b147
What licenses are recorded?
Code, training data, weights, and documents can have different terms. A blank declaration means unknown, not unrestricted use.
- Code
- Not recorded
- Data
- mit (publisher model/dataset card)
- Document
- Not recorded
- Weights
- Not recorded
Evaluation protocol
- Focus
- offline desktop/web action-script generation
- Setting
- offline
- Observations
- screenshot, natural-language task
- Actions / outputs
- PyAutoGUI script: pointer movement/click/drag, key presses, hotkeys, and text entry
- Evaluation
- The bundled evaluator reports weighted sequence match and action match, comparing action types and applying coordinate, key, and writing penalties. Target element-box metadata is evaluation-only. The retained scorer has a click-penalty defect: predicted coordinates are read from gold_line, so score reproduction requires explicit scorer-version review.
Scale and counting units
- 6,788 task examples — train split, HF revision 5e4bcb26 source
- 2,020 task examples — test split, HF revision 5e4bcb26 source
- 991 task examples — val split, HF revision 5e4bcb26 source
This released split table totals 9,799 examples; it is not silently substituted for any paper-level collection size. Public files include train/test/val JSON indexes, task/image archives, and eval.py. Evaluation is offline script comparison, not execution success on a live desktop. MIT is the Hub data declaration; original app/content rights are not resolved by it.
Compare source-reviewed profiles
Recorded specifications
- disabled
- false
- download url
- https://huggingface.co/datasets/Writer/omniact/tree/main
- gated
- false
- hub id
- Writer/omniact
- languages
- ["en"]
Explore related topics
Related resources
- documented by: OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web
Primary evidence and verification
Source retrieval is separate from experiment reproduction. Use the original publication or publisher page when citing claims about the resource.
- editorial primary review
Retrieved 2026-10-16T06:38:27Z. Supports: name, kind, summary, environments, topics, scope, relations, details.benchmark_protocol.
Content fingerprint
2db0a5dd2b983dded3b90a8ecd66cadbb39074f03778a339822418531661b3e3 - editorial primary review
Retrieved 2026-10-17T14:10:00Z. Supports: details.benchmark_protocol.
Content fingerprint
7b54ad557f89d621fe39c3d2fe9f02164a2bfcb7fef0db610d964586d3673693 - editorial primary review
Retrieved 2026-10-17T14:22:06Z. Supports: details.benchmark_protocol.
Content fingerprint
a786de678af46df5e2b6b1a2a69d2ee239fe8fd5b63663765d84d39e21fc2365 - huggingface api
Retrieved 2026-10-16T06:38:27Z. Supports: details, licenses.data.
Content fingerprint
2db0a5dd2b983dded3b90a8ecd66cadbb39074f03778a339822418531661b3e3
Suggest a correction
Open a source-backed correction. Include this record ID and the original source. Read our editorial policy and coverage boundaries.