AndroidLab
Reproducible Android benchmark of 138 tasks across nine apps on predefined virtual devices, run via AVD on Mac or Docker on Linux, released with the Android Instruct training set.
AI-assisted editorial review against a primary source. This is a catalog annotation, not an independent reproduction or endorsement.
What is this resource?
- Resource type
- Benchmarks
- Environment
- android
- Topics
- evaluation, mobile-navigation
- Record publication date
- Not recorded
- Evidence retrieved
- 2026-10-16T06:35:58Z
- Resource ID
- github:thudm/android-lab
What licenses are recorded?
Code, training data, weights, and documents can have different terms. A blank declaration means unknown, not unrestricted use.
- Code
- MIT (repository LICENSE header; not a weights/data license)
- Data
- Not recorded
- Document
- Not recorded
- Weights
- Not recorded
Evaluation protocol
- Focus
- Android task completion, subgoal completion, operation efficiency
- Setting
- interactive
- Observations
- UI XML, set-of-marks views
- Actions / outputs
- Android operations through the agent framework
- Evaluation
- The evaluation pipeline produces success, subgoal success, reversed redundancy, and reasonable-operation metrics; result generation requires a configured judge model.
Scale and counting units
The benchmark uses predefined virtual-device images and apps chosen to work without network/login requirements. XML and SoM modes should be reported separately. These counts concern evaluation tasks, not the separately released Android Instruct training set.
Compare source-reviewed profiles
Recorded specifications
- evaluation type
- benchmark task evaluation
- leaderboard url
- https://docs.google.com/spreadsheets/d/1Zv6mBfd4Ibt8mke24K6zAFrXe4AvEFaBLW3hpJqSgjw/edit?gid=0#gid=0
- metrics
- ["task success"]
- task count
- 138
Explore related topics
Related resources
Primary evidence and verification
Source retrieval is separate from experiment reproduction. Use the original publication or publisher page when citing claims about the resource.
- editorial primary review
Retrieved 2026-10-16T06:35:58Z. Supports: name, kind, summary, environments, topics, scope, relations, details.
Content fingerprint
340680a9a7833d103234b8367195deebd29bfec7e25ebc50c2dc04fc3627e55b - editorial primary review
Retrieved 2026-10-16T06:35:58Z. Supports: details.benchmark_protocol.
Content fingerprint
340680a9a7833d103234b8367195deebd29bfec7e25ebc50c2dc04fc3627e55b - license header
Retrieved 2026-10-16T06:41:58Z. Supports: licenses.code.
Content fingerprint
1bbbee919161337e2456c587c6be809214406482cd9f1292a9eb62428f6961f3
Suggest a correction
Open a source-backed correction. Include this record ID and the original source. Read our editorial policy and coverage boundaries.