Benchmarks

OS-Map

An OSWorld-derived desktop benchmark with an Ubuntu virtual-machine environment and expanded task configurations for measuring breadth and depth of computer interaction.

AI-assisted editorial review against a primary source. This is a catalog annotation, not an independent reproduction or endorsement.

Original source ↗Read as Markdown Record JSON

What is this resource?

Resource type: Benchmarks
Environment: desktop, linux
Topics: evaluation, long-horizon, cross-app
Record publication date: Not recorded
Evidence retrieved: 2026-10-17T09:12:14Z
Resource ID: github:os-copilot/os-map

What licenses are recorded?

Code, training data, weights, and documents can have different terms. A blank declaration means unknown, not unrestricted use.

Code: MIT (Repository code LICENSE; no license is inferred for the VM image or bundled third-party applications.)
Data: Not recorded
Document: Not recorded
Weights: Not recorded

License review notes

Code: Repository code LICENSE; no license is inferred for the VM image or bundled third-party applications. Source 1

Evaluation protocol

Focus: breadth of desktop task coverage, depth of desktop task execution
Setting: interactive
Observations: desktop observations from an Ubuntu virtual machine
Actions / outputs: actions executed by the OSWorld-derived DesktopEnv
Evaluation: Task configurations provide evaluators for application/system state, using an OSWorld-derived runtime.

Scale and counting units

416 category/task references — evaluation_examples/all.json (computed) source

416 category/task pairs are unique, but only 412 bare ID strings are unique across categories. This is a manifest-reference count, not a claim of 416 semantically distinct tasks. Main runtime documentation specifies Ubuntu; the subdirectory README retains older Windows-only wording.

Compare source-reviewed profiles

Recorded specifications

availability: Public task definitions, execution harness and an ungated VM-file manifest.
repository revision: 191efd0e6766acab444a31685b437565e1a94ac2

Explore related topics

Evaluation Long-horizon tasks Cross-app

Scope and limitations

Public task manifests, agent runner and evaluator code are present, and the linked ungated Hub manifest lists Ubuntu0.zip and task files. The main setup supports x86 VMware Workstation and says ARM/Docker alternatives are still in development. An inherited evaluation-examples README still mentions Windows 10 and under-construction status; those statements are retained as a documentation inconsistency, not interpreted as verified Windows support. The manifest has 416 category/task references but 412 globally distinct ID strings.

Related resources

uses: OSWorld
documented by: OS-MAP: How Far Can Computer-Using Agents Go in Breadth and Depth?

Primary evidence and verification

Source retrieval is separate from experiment reproduction. Use the original publication or publisher page when citing claims about the resource.

editorial primary review
Retrieved 2026-10-17T09:12:14Z. Supports: name, kind, summary, environments, topics, scope, notes, relations, details.
Content fingerprint
fc9f3c8d9fffdff8b51e312b013be0d9fdfc10e57fdc2ec4a8a8d012c4b42475
editorial primary review
Retrieved 2026-10-17T09:12:14Z. Supports: identity.
Content fingerprint
d6c91b4910e92f40cc694b4cb6467c1a20f42c0c14047b95a1e777aca0892677
editorial primary review
Retrieved 2026-10-17T09:14:07Z. Supports: release_status.
Content fingerprint
41342157c53f28ad2729137491930225ac11da922e7b16762e84ac958a2e9d28
editorial primary review
Retrieved 2026-10-17T09:14:07Z. Supports: licenses.code.
Content fingerprint
1dd924135cbf0a85ce72d75025259adb7b849970d400af4ed854327a34e7ee41
editorial primary review
Retrieved 2026-10-17T09:17:22Z. Supports: details.benchmark_protocol.
Content fingerprint
4199af4c8bc60451079d8a2e7598ab7386bd82b0d5c9ad7e6927d6cb8e03ad6f
editorial primary review
Retrieved 2026-10-17T09:17:22Z. Supports: details.benchmark_protocol, notes.
Content fingerprint
a6b60d5721cfdd26416e7d0ae3192360896427a97ccb3793812cc150e9cd7126
editorial primary review
Retrieved 2026-10-17T09:14:16Z. Supports: release_status, details.
Content fingerprint
d2da837872b123f2927a069a020aeb14fb4b22148599c9f2e8b6689ee444ec88

Suggest a correction

Open a source-backed correction. Include this record ID and the original source. Read our editorial policy and coverage boundaries.