AwesomeCUAGitHub ↗
Benchmarks

ColorBench

A graph-based mobile GUI benchmark that evaluates long tasks through screenshot states and multiple valid or erroneous transitions, including reflection, replanning and backtracking.

AI-assisted editorial review against a primary source. This is a catalog annotation, not an independent reproduction or endorsement.

Original source ↗Read as MarkdownRecord JSON

What is this resource?

Resource type
Benchmarks
Environment
android
Topics
evaluation, mobile-navigation, long-horizon, cross-app
Record publication date
Not recorded
Evidence retrieved
2026-10-17T11:18:16Z
Resource ID
github:madeagents/colorbench

What licenses are recorded?

Code, training data, weights, and documents can have different terms. A blank declaration means unknown, not unrestricted use.

Code
Apache-2.0 (Repository source LICENSE.)
Data
apache-2.0 (Publisher metadata for the separately hosted screenshot archive; underlying third-party content retains applicable rights.)
Document
Not recorded
Weights
Not recorded

License review notes

Evaluation protocol

Focus
long-horizon mobile task completion, multi-path graph interaction
Setting
interactive
Observations
mobile screenshots represented as graph states
Actions / outputs
mobile GUI operations along graph transitions
Evaluation
The publisher reports success rate, completion rate and atomic-capability diagnostics using graph-based evaluation.

Scale and counting units

The interaction environment replays graph states rather than running real user accounts. The 101 and 74 subsets partition the reported 175 tasks; do not add the three counts.

Compare source-reviewed profiles

Recorded specifications

availability
Public graph/task manifests, evaluation source and ungated screenshot archive manifest.
repository revision
1f26a84230d26f99d84eab97d05032582b39018e

Explore related topics

EvaluationMobile navigationLong-horizon tasksCross-app

Scope and limitations

The source tree contains graph and task JSON files plus construction/evaluation code; the linked ungated Hub provides the screenshot archive. The environment is static but interactive and graph-based, not a live-device deployment. Counts are publisher-reported task/app inventories. Apache-2.0 declarations for repository code and the Hub dataset are separately retained; third-party application visuals are not granted additional rights by this catalog.

Related resources

Primary evidence and verification

Source retrieval is separate from experiment reproduction. Use the original publication or publisher page when citing claims about the resource.

  1. editorial primary review

    Retrieved 2026-10-17T11:18:16Z. Supports: name, kind, summary, environments, topics, scope, notes, relations, details.benchmark_protocol.

    Content fingerprint5c0190fffb68b91c1855a408f6a5e252871d94000b95fa258504a3c100600cec
  2. editorial primary review

    Retrieved 2026-10-17T11:18:16Z. Supports: identity.

    Content fingerprint2f6940503cc0d6cb97bdf142d6b8c6ab116de82227f2fa6d97fd56d7e5aebec0
  3. editorial primary review

    Retrieved 2026-10-17T11:22:54Z. Supports: release_status, details.repository_revision.

    Content fingerprint6a00c093a6794ec4a172d30b70fdabc17400584a9e25a157a47a7661dbdd20fa
  4. editorial primary review

    Retrieved 2026-10-17T11:23:06Z. Supports: identity, release_status.

    Content fingerprintc0c3286b5150e128e55034ba0393ef85f7fcf73e4b73f4276389a04c9bcec919
  5. editorial primary review

    Retrieved 2026-10-17T11:23:06Z. Supports: licenses.data.

    Content fingerprint4bcf87ecfbbb8e07a01b21415a970c8b53a5283bf6872b657040d3f45c9241f7
  6. editorial primary review

    Retrieved 2026-10-17T11:22:54Z. Supports: licenses.code.

    Content fingerprint130976c026bcafa9dcc479697fe6595dbefb73cdb375fc47b8478acc16e6ea73

Suggest a correction

Open a source-backed correction. Include this record ID and the original source. Read our editorial policy and coverage boundaries.