SPA-Bench

340 single-app and cross-app tasks over 66 Chinese and English Android apps, with plug-in support for 11 smartphone agents and automatic coarse-to-fine success detection.

AI-assisted editorial review against a primary source. This is a catalog annotation, not an independent reproduction or endorsement.

What is this resource?

Resource type

Benchmarks

Environment

android

Topics

evaluation, mobile-navigation

Record publication date

Not recorded

Evidence retrieved

2026-10-16T06:35:58Z

Resource ID

github:ai-agents-2030/spa-bench

What licenses are recorded?

Code, training data, weights, and documents can have different terms. A blank declaration means unknown, not unrestricted use.

Code

MIT (repository LICENSE header; not a weights/data license)

Data

Not recorded

Document

Not recorded

Weights

Not recorded

Evaluation protocol

Focus

mobile task completion, resource use, cross-app workflows

Setting

interactive

Observations

screenshot trajectories

Actions / outputs

agent-specific Android actions

Evaluation

A multimodal evaluator and annotated key-component matching assess completion; the framework also reports resource-use metrics.

Scale and counting units

• 340 tasks — complete README task collection source
• 300 tasks — single-app subset source
• 40 tasks — cross-app subset source
• 66 applications — README total including third-party, Google, and system apps source

The two task subsets partition the 340 tasks. The integrated agent and evaluator modes must accompany results. The preconfigured emulator snapshot is still marked coming soon in the inspected documentation.

Compare source-reviewed profiles

Explore related topics

Related resources

• documented by: SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation

Primary evidence and verification

Source retrieval is separate from experiment reproduction. Use the original publication or publisher page when citing claims about the resource.

1. editorial primary review

   Retrieved 2026-10-16T06:35:58Z. Supports: name, kind, summary, environments, topics, scope, relations, details.

   Content fingerprint

   5294a67728072a3658cb2073bdf9a084fe3c00d87a848923dff68adb35213072
2. editorial primary review

   Retrieved 2026-10-16T06:35:58Z. Supports: details.benchmark_protocol.

   Content fingerprint

   5294a67728072a3658cb2073bdf9a084fe3c00d87a848923dff68adb35213072
3. editorial primary review

   Retrieved 2026-10-17T02:03:23Z. Supports: details.benchmark_protocol.

   Content fingerprint

   a55290d62f8823097c73d74bac4eb08853a3caff724382e60a6a9364e7ced987
4. license header

   Retrieved 2026-10-16T06:41:58Z. Supports: licenses.code.

   Content fingerprint

   6962bfc8ee16c6e985e4fb680b8fe46d5b34d30f7cb8a1e61fd4c55c93120ddf

Suggest a correction

Open a source-backed correction. Include this record ID and the original source. Read our editorial policy and coverage boundaries.