WindowsAgentArena-V2

Windows benchmark revision with task/evaluator fixes and VM-initialization support, documented alongside PC-Agent-E.

AI-assisted editorial review against a primary source. This is a catalog annotation, not an independent reproduction or endorsement.

What is this resource?

Resource type

Benchmarks

Environment

desktop, windows

Topics

evaluation, cross-app, robustness

Record publication date

Not recorded

Evidence retrieved

2026-10-18T04:35:17Z

Resource ID

github:gair-nlp/windowsagentarena-v2

What licenses are recorded?

Code, training data, weights, and documents can have different terms. A blank declaration means unknown, not unrestricted use.

Code

MIT (Reviewed repository license declaration for code; third-party dependencies, data, weights and executable environment images are separate.)

Data

Not recorded

Document

Not recorded

Weights

Not recorded

License review notes

• Code: Reviewed repository license declaration for code; third-party dependencies, data, weights and executable environment images are separate. Source 1

Evaluation protocol

Focus

Windows desktop task execution, revised task evaluators

Setting

interactive

Observations

screenshot, accessibility tree, screenshot plus accessibility tree, set-of-marks

Actions / outputs

PyAutoGUI

Evaluation

Runner exposes four observation choices and uses PyAutoGUI. count.py sums the last score in each results.json and divides by the number of traj.jsonl files, multiplying by 100; it separately reports result/trajectory/error-file coverage. This denominator is not automatically the full benchmark size.

Scale and counting units

• 141 tasks — full WindowsAgentArena-V2 suite reported by README source
• 11 Windows applications — full suite coverage reported by README source
• 20 task IDs — shipped test_custom.json selection (computed) source

README sample config uses max_steps=30. The shipped selected-task index has 20 IDs, not 141; preserve the selected task list, trial ID, score coverage and VM snapshot when comparing runs. Docker/Windows-11 setup is required. No VM or benchmark was executed.

Compare source-reviewed profiles

Recorded specifications

repository tree sha

2927fe55005d1be75d5a9188c0045f73ba28d192

repository url

https://github.com/GAIR-NLP/WindowsAgentArena-V2

Explore related topics

Scope and limitations

Repository contains executable evaluation code and task definitions. README links a separate Windows 11 VM snapshot; that image and its software licenses/access were not validated by downloading it. MIT is recorded for repository code only. Default task selection and the reported full suite have different scopes.

Related resources

• documented by: Efficient Agent Training for Computer Use

Primary evidence and verification

Source retrieval is separate from experiment reproduction. Use the original publication or publisher page when citing claims about the resource.

1. editorial primary review

   Retrieved 2026-10-18T04:35:17Z. Supports: name, kind, summary, environments, topics, scope, notes, relations, details.benchmark_protocol.

   Content fingerprint

   1dd744ce565ec00b20ea5637eb7bc6112e97fb9044bd9a471f7cc4b7227ca344
2. editorial primary review

   Retrieved 2026-10-18T04:33:20Z. Supports: identity.

   Content fingerprint

   0b0af02d3c2c4916dd786b868b99d60b4681927d0273a23661eb2b6db7cf9404
3. editorial primary review

   Retrieved 2026-10-18T04:35:16Z. Supports: identity, release_status, details.repository_tree_sha, details.benchmark_protocol.

   Content fingerprint

   638ef9ee593a3a560c1d0b3fb2bc4a9d31a3e904c0111971924a6d66e1763aa6
4. editorial primary review

   Retrieved 2026-10-18T04:40:39Z. Supports: details.benchmark_protocol.

   Content fingerprint

   494de9e74629ac2c55358629c0d9ba13703f36317057a3089388fbb6e80ca28f
5. editorial primary review

   Retrieved 2026-10-18T04:40:39Z. Supports: details.benchmark_protocol.

   Content fingerprint

   e45ba7fdc7555f46ad1b4cfaab79d00754909d053fc49d5bab651b0fb593439b
6. editorial primary review

   Retrieved 2026-10-18T04:40:39Z. Supports: details.benchmark_protocol.

   Content fingerprint

   f12147ac78c39c72eb75f9c37b293ba7fff3209fc5758fd51c69abbd96d80a97
7. editorial primary review

   Retrieved 2026-10-18T04:35:20Z. Supports: licenses.code.

   Content fingerprint

   2a2d1e46940b570d60f5e2198ed27fcec6316fc8489e9660da8777dbd75aaeec

Suggest a correction

Open a source-backed correction. Include this record ID and the original source. Read our editorial policy and coverage boundaries.