OS-Harm

Benchmark for safety of computer-use agents across misuse, prompt injection, and unintended model behavior.

AI-assisted editorial review against a primary source. This is a catalog annotation, not an independent reproduction or endorsement.

What is this resource?

Resource type

Benchmarks

Environment

desktop

Topics

safety-security, evaluation

Record publication date

Not recorded

Evidence retrieved

2026-10-16T09:36:33Z

Resource ID

github:tml-epfl/os-harm

What licenses are recorded?

Code, training data, weights, and documents can have different terms. A blank declaration means unknown, not unrestricted use.

Code

Apache-2.0 (repository LICENSE header; not a weights/data license)

Data

Not recorded

Document

Not recorded

Weights

Not recorded

Evaluation protocol

Focus

deliberate misuse, prompt-injection safety, unintended harmful behavior during normal tasks

Setting

interactive

Observations

screenshot, accessibility tree, screenshot plus accessibility tree, set-of-marks

Actions / outputs

OSWorld/PyAutoGUI desktop interaction

Evaluation

Automated LLM judge scores both task accuracy and safety from saved execution traces. Keep judge model/prompt version and harm category with a result; the scripts separately compute predicted metrics. This is an interactive desktop safety benchmark, not a single offline grounding score.

Scale and counting units

• 150 tasks as reported by the paper/README — reported benchmark total; not a recount of unique released task IDs source
• 50 task IDs — released misuse task-ID list; before injection/goal expansion (computed) source
• 10 task IDs — released injection task-ID list; before injection/goal expansion (computed) source
• 50 task IDs — released misbehavior task-ID list; before injection/goal expansion (computed) source

README/paper reports 150 tasks, whereas retained misuse/injection/misbehavior lists contain 50/10/50 IDs. run.py expands injection vectors/goals, so the released ID inventory is not a direct recount of paper settings; the complete expanded total was not reconstructed. Judge agreement F1 values in the README measure agreement with human labels, not agent success. Experiments use OSWorld desktop apps, with VMware documented as tested and screenshot/a11y/SOM observation choices. Code Apache-2.0 does not establish VM/app/data rights. Publisher asks that the benchmark not be used directly as agent training data.

Compare source-reviewed profiles

Recorded specifications

evaluation type

automated judgment of accuracy and safety

metrics

["task completion", "safety"]

task count

150

task unit

tasks reported by the paper/README; not a recount of released task IDs

Explore related topics

Scope and limitations

The retained task-ID manifests have 50 misuse, 10 injection and 50 misbehavior IDs. Prompt-injection execution expands vectors/goals; the reported 150-task total must not be read as 150 unique released IDs.

Related resources

• documented by: OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents

Primary evidence and verification

Source retrieval is separate from experiment reproduction. Use the original publication or publisher page when citing claims about the resource.

1. editorial primary review

   Retrieved 2026-10-16T09:36:33Z. Supports: name, kind, summary, environments, topics, scope, relations, details, details.benchmark_protocol, notes.

   Content fingerprint

   d001158f58ac37b01c8470224db1567a52feb8010877305f7966637cf7400706
2. editorial primary review

   Retrieved 2026-10-17T14:23:14Z. Supports: details.benchmark_protocol, notes.

   Content fingerprint

   86dd59a252fd654fc3244268a75de37b2c7f97e16f6b1de8c4cf885458a422f7
3. editorial primary review

   Retrieved 2026-10-17T14:22:04Z. Supports: details.benchmark_protocol, notes.

   Content fingerprint

   8ff6092a51cc924c5225ace01c011c6481a2461eb0a47b7360eddaf6619b898c
4. editorial primary review

   Retrieved 2026-10-17T14:22:04Z. Supports: details.benchmark_protocol, notes.

   Content fingerprint

   b4c451236bb4ff36eb7e790dfa8cb517bffb245602bc441080314b13c8aba54f
5. editorial primary review

   Retrieved 2026-10-17T14:22:05Z. Supports: details.benchmark_protocol, notes.

   Content fingerprint

   8eb6bade72696c3c0b53160a05b006a773c9e553ab5e401418b54d9acc98c0f2
6. license header

   Retrieved 2026-10-16T09:39:34Z. Supports: licenses.code.

   Content fingerprint

   7c3ab1eeeb073d92238ecb65ffaebe58ab5922929b115a7c0b64eb4f9a9afc2f

Suggest a correction

Open a source-backed correction. Include this record ID and the original source. Read our editorial policy and coverage boundaries.