AwesomeCUAGitHub ↗
Benchmarks

MobileSafetyBench

Android emulator testbed that scores mobile device-control agents on both safety and helpfulness, using ADB and Appium for observations, actions and success checks.

AI-assisted editorial review against a primary source. This is a catalog annotation, not an independent reproduction or endorsement.

Original source ↗Read as MarkdownRecord JSON

What is this resource?

Resource type
Benchmarks
Environment
android
Topics
safety-security, evaluation
Record publication date
Not recorded
Evidence retrieved
2026-10-16T06:35:58Z
Resource ID
github:jylee425/mobilesafetybench

What licenses are recorded?

Code, training data, weights, and documents can have different terms. A blank declaration means unknown, not unrestricted use.

Code
Apache-2.0 (repository LICENSE header; not a weights/data license)
Data
Not recorded
Document
Not recorded
Weights
Not recorded

Evaluation protocol

Focus
mobile task helpfulness, harm prevention during Android interaction
Setting
interactive
Observations
screenshot pixels, parsed UI hierarchy text, task instruction and interaction history
Actions / outputs
tap/long-tap/swipe, navigation buttons and text editing, refuse, ask-consent and complete, task-specific shortcut actions
Evaluation
Appium-backed task evaluators track goal achievement and harm prevention as separate booleans, plus risk-detection step and completion progress. The base evaluator defaults to max_step=15; task-specific overrides must be retained. Results are not a single generic success-rate metric.

Scale and counting units

Not recorded

Inspected release/ver.3 branch exposes task definitions, Android environment and per-task evaluators. It requires Android API 34, Appium/ADB and documented third-party app versions; those APKs/accounts were not acquired. The action parser includes semantic shortcuts in addition to screen gestures, so this is not a strictly coordinate-only action setting. No corpus task total is inferred from evaluator-module or file counts. Apache-2.0 applies to repository code, not automatically app/data assets.

Compare source-reviewed profiles

Explore related topics

Safety & securityEvaluation

Primary evidence and verification

Source retrieval is separate from experiment reproduction. Use the original publication or publisher page when citing claims about the resource.

  1. editorial primary review

    Retrieved 2026-10-16T06:35:58Z. Supports: name, kind, summary, environments, topics, scope, relations, details.benchmark_protocol.

    Content fingerprinte733115d9b2fbe910cb3891162f04a89af8cbda25255c8f0bf60f78101e4309a
  2. editorial primary review

    Retrieved 2026-10-17T14:14:05Z. Supports: details.benchmark_protocol.

    Content fingerprint8dc66f6c5087a32c5c9fb7b3104ea6f9e0082255e2ba3cf9ba8244ce5a24da6f
  3. editorial primary review

    Retrieved 2026-10-17T14:21:09Z. Supports: details.benchmark_protocol.

    Content fingerprintf56d6345546a9c67a2113cfba5b7148f5bd6b48fac5c34f1a5f4080e2d81ae2f
  4. editorial primary review

    Retrieved 2026-10-17T14:21:10Z. Supports: details.benchmark_protocol.

    Content fingerprint120487c3a6492371321dd3eebcd65dc59de6a82454d622d7fed3a7d86b8fa6c5
  5. editorial primary review

    Retrieved 2026-10-17T14:23:14Z. Supports: details.benchmark_protocol.

    Content fingerprint918e3aa64ab3f578fb54580da123c1ff4508022b64ce74f88ce28c7a612756a4
  6. license header

    Retrieved 2026-10-16T06:41:58Z. Supports: licenses.code.

    Content fingerprintcfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30

Suggest a correction

Open a source-backed correction. Include this record ID and the original source. Read our editorial policy and coverage boundaries.