Benchmarks

OS-Critic Bench

A cross-platform GUI-critic benchmark with mobile, desktop and web JSONL splits and screenshot assets, accompanied by public inference and accuracy-scoring scripts.

AI-assisted editorial review against a primary source. This is a catalog annotation, not an independent reproduction or endorsement.

Original source ↗Read as Markdown Record JSON

What is this resource?

Resource type: Benchmarks
Environment: cross-platform, desktop, web
Topics: evaluation, reward-models, screen-understanding
Record publication date: Not recorded
Evidence retrieved: 2026-10-17T11:18:18Z
Resource ID: url:c73ba0618d1b3c505887

What licenses are recorded?

Code, training data, weights, and documents can have different terms. A blank declaration means unknown, not unrestricted use.

Code: Not recorded
Data: apache-2.0 (Publisher dataset-card license declaration for OS-Critic-Bench; not an evaluation-code or model-weight license.)
Document: Not recorded
Weights: Not recorded

License review notes

Data: Publisher dataset-card license declaration for OS-Critic-Bench; not an evaluation-code or model-weight license. Source 1

Evaluation protocol

Focus: GUI critic prediction
Setting: offline
Observations: benchmark JSONL, UI screenshots
Actions / outputs: critic-model responses
Evaluation: The official repository runs model inference and computes accuracy with cal_acc.py.

Scale and counting units

3 platform JSONL splits — mobile/desktop/web configurations in the Hub card (computed) source

No sample count is inferred from the n<1K Hub size band. Evaluation code availability does not establish the release of OS-Oracle training data or weights.

Compare source-reviewed profiles

Recorded specifications

availability: Ungated benchmark JSONL/image manifest and public evaluation scripts; training pipeline/checkpoints are pending.
gated: false
hub id: OS-Copilot/OS-Critic-Bench
release revision: 8fe774ef6a422c0e555f90205eb392f9132e272b

Explore related topics

Evaluation Reward models Screen understanding

Scope and limitations

The Hub card provides split/format/license metadata, while the official OS-Oracle README establishes critic-evaluation purpose and platform scope. The repository currently releases only benchmark inference/scoring components; data-synthesis code, training datasets and model checkpoints remain unchecked TODOs and are not counted as released assets. Apache-2.0 is recorded for benchmark data only; the evaluation code has no inspected license declaration.

Related resources

documented by: OS-Oracle: A Comprehensive Framework for Cross-Platform GUI Critic Models

Primary evidence and verification

Source retrieval is separate from experiment reproduction. Use the original publication or publisher page when citing claims about the resource.

editorial primary review
Retrieved 2026-10-17T11:18:18Z. Supports: name, kind, summary, environments, topics, scope, notes, relations, details.benchmark_protocol.
Content fingerprint
58305b8ebe1af5dfbddd636484d63722d4e21d28ed3e22bb8c0068e654f33479
editorial primary review
Retrieved 2026-10-17T11:18:17Z. Supports: identity.
Content fingerprint
17bcf851f39d06e7e357871df1dc1ddf697bcbea31842c16348227bbf4fbb224
editorial primary review
Retrieved 2026-10-17T11:22:55Z. Supports: release_status, licenses.code.
Content fingerprint
516beea465382b54548e4b8bfd12ae510080a49451b30262ad1c769ed5a3398e
editorial primary review
Retrieved 2026-10-17T11:23:06Z. Supports: identity, release_status, details.hub_id, details.gated, details.release_revision.
Content fingerprint
ef311b49437e246ffef6ad13c090277a0bdacc4a124e5b3cfe8d2643ced8f843
editorial primary review
Retrieved 2026-10-17T11:23:07Z. Supports: licenses.data, details.benchmark_protocol.
Content fingerprint
4139e25e8911ca8450f6db9acb8dc3a98c126c3c43514c72bcefa4ed33b8ffa8

Suggest a correction

Open a source-backed correction. Include this record ID and the original source. Read our editorial policy and coverage boundaries.