ClawBench
Everyday-task benchmark on live websites that runs browser agents in containerized Chromium, blocks each final submission request for scoring by URL match and an LLM judge, and records five layers of run evidence.
AI-assisted editorial review against a primary source. This is a catalog annotation, not an independent reproduction or endorsement.
What is this resource?
- Resource type
- Benchmarks
- Environment
- web
- Topics
- evaluation, web-navigation, workflow, observability
- Record publication date
- Not recorded
- Evidence retrieved
- 2026-10-18T11:53:46Z
- Resource ID
- github:tiger-ai-lab/clawbench
What licenses are recorded?
Code, training data, weights, and documents can have different terms. A blank declaration means unknown, not unrestricted use.
- Code
- Apache-2.0 (repository LICENSE header; not a weights/data license)
- Data
- Not recorded
- Document
- Not recorded
- Weights
- Not recorded
Evaluation protocol
- Focus
- everyday online tasks on production websites, multi-step, write-heavy form workflows
- Setting
- interactive
- Observations
- Not recorded
- Actions / outputs
- harness-defined browser actions such as navigating, clicking and filling forms
- Evaluation
- Two stages per task run: a request interceptor inside the container blocks the final HTTP request and marks the run intercepted when its URL regex and method match the task's eval_schema; an LLM judge (default deepseek-v4-pro at temperature 0) then checks the intercepted body against the instruction. A pass needs both. The original V1 paper rubric instead compares agent and human-reference trajectories.
Scale and counting units
- 152 tasks — V1 corpus shipped in test-cases/v1 (task.json files) (computed) source
- 129 tasks — V2 corpus shipped in test-cases/v2 (task.json files) (computed) source
- 153 tasks — V1 corpus as published in the paper and used as the scoring denominator source
- 130 tasks — V2 corpus as published and used as the scoring denominator source
- 143 live websites — shipped V1 corpus source
- 63 live websites — shipped V2 corpus source
The README says two ASPCA tasks were removed after publication, so the shipped corpora hold 152 and 129 tasks while scoring.md still divides by 153 and 130. Live sites change over time, and the README asks for comparisons under the same corpus revision, harness and rubric. Agents run through selectable harnesses (for example Hermes and OpenClaw), so observations and actions depend on the harness.
Compare source-reviewed profiles
Recorded specifications
- leaderboard url
- https://huggingface.co/spaces/TIGER-Lab/ClawBench
- repository revision
- 07eb50a3d4c5a799f5ad4eebd57c67c10417787f
- repository tree sha
- d12a2dda577f2c943a74d833e8253860eda81eaf
Explore related topics
Scope and limitations
Renamed from reacher-z/ClawBench; both API names return repository 1206519531 (TIGER-AI-Lab/ClawBench), which is not a fork and not archived. Counts and scoring rules were read at commit 07eb50a3. The tree also holds v1-lite, claw-eval and environmental-observation task folders, which are not counted here. Task definitions and execution traces are also published as Hugging Face datasets (NAIL-Group/ClawBench, NAIL-Group/ClawBenchV1Trace, TIGER-Lab/ClawBenchV2Trace), whose cards and licenses were not reviewed for this record. Apache-2.0 is recorded for repository code only. Nothing was installed or run.
Related resources
- documented by: ClawBench: Can AI Agents Complete Everyday Online Tasks?
Primary evidence and verification
Source retrieval is separate from experiment reproduction. Use the original publication or publisher page when citing claims about the resource.
- editorial primary review
Retrieved 2026-10-18T11:53:46Z. Supports: name, kind, summary, environments, topics, scope, notes, relations, details.
Content fingerprint
61bad2a8e6ab56df159237d75f1081e1246b294183b31ef7abe47193ac26b541 - editorial primary review
Retrieved 2026-10-18T11:54:35Z. Supports: details.repository_revision, details.repository_tree_sha, details.benchmark_protocol.
Content fingerprint
144a65c3dd92c7437e4ce065bed7b16e4c33496ea395d2bb1ecbd74132da29ce - editorial primary review
Retrieved 2026-10-18T11:54:36Z. Supports: details.benchmark_protocol.
Content fingerprint
8d8ade83a355e10800591435de5aa111e8d8297374b025ff88804358ee80b71a - license header
Retrieved 2026-10-18T11:55:00Z. Supports: licenses.code.
Content fingerprint
cb861e529412e9eec01498bdf4b74c5ee4f29139737a1193e39b24004019e606 - github repository identity
Retrieved 2026-10-18T11:54:36Z. Supports: aliases.
Content fingerprint
4b80afa11f3998830090c9c186afc753f261cc05abbcd4d906400f757f41e4da - github repository identity
Retrieved 2026-10-18T11:54:37Z. Supports: aliases.
Content fingerprint
4b80afa11f3998830090c9c186afc753f261cc05abbcd4d906400f757f41e4da
Suggest a correction
Open a source-backed correction. Include this record ID and the original source. Read our editorial policy and coverage boundaries.