Benchmarks

OpenApps

Configurable browser apps and task rewards for testing computer-use agents under changes in content, layout and appearance.

AI-assisted editorial review against a primary source. This is a catalog annotation, not an independent reproduction or endorsement.

Original source ↗Read as Markdown Record JSON

What is this resource?

Resource type: Benchmarks
Environment: web
Topics: evaluation, robustness, cross-app, sandbox
Record publication date: Not recorded
Evidence retrieved: 2026-10-18T04:35:18Z
Resource ID: github:facebookresearch/openapps

What licenses are recorded?

Code, training data, weights, and documents can have different terms. A blank declaration means unknown, not unrestricted use.

Code: CC-BY-NC-4.0 (Reviewed repository license declaration for code; third-party dependencies, data, weights and executable environment images are separate.)
Data: Not recorded
Document: Not recorded
Weights: Not recorded

License review notes

Code: Reviewed repository license declaration for code; third-party dependencies, data, weights and executable environment images are separate. Source 1

Evaluation protocol

Focus: UI-agent reliability under environment variation, state-based browser task evaluation
Setting: interactive
Observations: screenshots, accessibility trees, set-of-marks and coordinates (configurable)
Actions / outputs: click, type, scroll, configured BrowserGym/AgentLab actions
Evaluation: Tasks define check_if_task_is_complete(initial_state,current_state); app-specific reward logic evaluates underlying state, with custom checks for code-editor file contents. Observation/history flags are configurable. No universal single fixed task total or published-run reproduction is implied.

Scale and counting units

Not recorded

Preserve the code/tag, app state, theme/layout/content choices, goal variant and action/observation configuration. Current main refactors the paper-era appearance surface, and the v1.0-paper tag includes post-publication fixes that may move rewards. The environments run as browser apps; phone/tablet viewports are not native Android/iOS environments.

Compare source-reviewed profiles

Recorded specifications

repository tree sha: 6e6782794433c60468d65ac75bc3f03c56007f8c
repository url: https://github.com/facebookresearch/OpenApps

Explore related topics

Evaluation Robustness Cross-app Sandboxes

Scope and limitations

Public code includes app/task/configuration and agent-launching implementations. README distinguishes moving main from v1.0-paper and explicitly says even that tag is not a byte-exact snapshot of the published runs. CC-BY-NC-4.0 is the repository’s declared code license; third-party WebShop, map and icon content retain separate terms. No commercial-use permission or fixed downloadable task total is inferred.

Related resources

documented by: OpenApps: Simulating Environment Variations to Measure UI-Agent Reliability

Primary evidence and verification

Source retrieval is separate from experiment reproduction. Use the original publication or publisher page when citing claims about the resource.

editorial primary review
Retrieved 2026-10-18T04:35:18Z. Supports: name, kind, summary, environments, topics, scope, notes, relations, details.benchmark_protocol.
Content fingerprint
1df0f66584b37e7dba90811a7732d3b0c13c96cd7da4aec3970a18be36a6cec1
editorial primary review
Retrieved 2026-10-18T04:33:19Z. Supports: identity.
Content fingerprint
2c57b627cb344e360480c10e74dcdd1d1ed321d9a7a4e3891bc970355b15f4cd
editorial primary review
Retrieved 2026-10-18T04:35:18Z. Supports: identity, release_status, details.repository_tree_sha, details.benchmark_protocol.
Content fingerprint
34aa76fbd6cca6ac2f3949792144662b90d85d7a18988afabcb570167ea5c1d6
editorial primary review
Retrieved 2026-10-18T04:40:40Z. Supports: details.benchmark_protocol.
Content fingerprint
af030b93a7604c063b08f003b0a0124a6e95e560649d0306bfe40dd87fed563d
editorial primary review
Retrieved 2026-10-18T04:40:40Z. Supports: details.benchmark_protocol.
Content fingerprint
aede4dfd965bd3484336f40ddb1bd6db48aed00ab45e27c048b76cc4601d71b3
editorial primary review
Retrieved 2026-10-18T04:35:20Z. Supports: licenses.code.
Content fingerprint
336255dc30193e8e15d689d9481bb05673d89055718f3a96923a7ffb99adbbaf

Suggest a correction

Open a source-backed correction. Include this record ID and the original source. Read our editorial policy and coverage boundaries.