Benchmarks

AndroidArena

Android environment for evaluating language-model-driven agents.

AI-assisted editorial review against a primary source. This is a catalog annotation, not an independent reproduction or endorsement.

Original source ↗Read as Markdown Record JSON

What is this resource?

Resource type: Benchmarks
Environment: android
Topics: evaluation, mobile-navigation
Record publication date: Not recorded
Evidence retrieved: 2026-10-16T06:35:58Z
Resource ID: github:androidarenaagent/androidarena

What licenses are recorded?

Code, training data, weights, and documents can have different terms. A blank declaration means unknown, not unrestricted use.

Code: Not recorded
Data: Not recorded
Document: Not recorded
Weights: Not recorded

Evaluation protocol

Focus: multi-step Android tasks, cross-app workflows and instruction constraints
Setting: interactive
Observations: UI hierarchy text, screenshots, combined text/image observations
Actions / outputs: app launch/stop, click/double-click/long-click, text input, swipe/drag and Android navigation, finish
Evaluation: The released evaluator compares trajectories with reference actions using longest-common-subsequence-based task reward and completion. Its reversed redundancy metric is the reference/predicted action-length ratio (with 1e-6 denominator smoothing), multiplied by the LLM success flag. LLM success judgments and instruction-constraint checks are separate metrics. Full reference-dependent evaluation coverage cannot be assumed because complete annotated action sequences are not released.

Scale and counting units

Not recorded

README states that task instructions are released but annotated action sequences, apart from the calendar example, will come later. Environment setup targets Android API 33 with uiautomator2 and manual Google-account setup. Public task YAML and evaluator code exist, but benchmark execution and full reference coverage were not verified. No task total is derived from app YAML file counts. Code and data licenses remain unknown.

Compare source-reviewed profiles

Explore related topics

Evaluation Mobile navigation

Related resources

documented by: Understanding the Weakness of Large Language Model Agents within a Complex Android Environment

Primary evidence and verification

Source retrieval is separate from experiment reproduction. Use the original publication or publisher page when citing claims about the resource.

editorial primary review
Retrieved 2026-10-16T06:35:58Z. Supports: name, kind, summary, environments, topics, scope, relations, details.benchmark_protocol.
Content fingerprint
206bd0dc5220cf754cffc97767cce6a3cd27660ef050a569366bef2331241c60
editorial primary review
Retrieved 2026-10-17T14:21:10Z. Supports: details.benchmark_protocol.
Content fingerprint
a1e7dc0c5facae030c2cf19aad6511996e7e1198d8d6619268d10c29a1b57bcf
editorial primary review
Retrieved 2026-10-17T14:14:05Z. Supports: details.benchmark_protocol.
Content fingerprint
edc03388b67143644db4a412e4c888f8c796068daff29f63615f0b6a3a04262a
editorial primary review
Retrieved 2026-10-17T14:24:52Z. Supports: details.benchmark_protocol.
Content fingerprint
cfa9d272c85398f54c18d6e7445d680009e8c16d128962235d899592d3094c2f
editorial primary review
Retrieved 2026-10-17T14:24:53Z. Supports: details.benchmark_protocol.
Content fingerprint
d25f2edf9202f59615341b62fc6046ac23a6f4cb072a6038ac562506cbdc891c
editorial primary review
Retrieved 2026-10-17T14:21:10Z. Supports: details.benchmark_protocol.
Content fingerprint
257c42e980a0b14bd74bfcc381a0ecbef2b538c0f1cc1a94d22925efe354fb8e

Suggest a correction

Open a source-backed correction. Include this record ID and the original source. Read our editorial policy and coverage boundaries.