AgentRewardBench

A benchmark and evaluation toolkit for comparing automatic web-agent judges against expert judgments of task success, side effects and repetition.

AI-assisted editorial review against a primary source. This is a catalog annotation, not an independent reproduction or endorsement.

What is this resource?

Resource type

Benchmarks

Environment

web

Topics

evaluation, reward-models, web-navigation

Record publication date

Not recorded

Evidence retrieved

2026-10-17T03:59:54Z

Resource ID

github:mcgill-nlp/agent-reward-bench

What licenses are recorded?

Code, training data, weights, and documents can have different terms. A blank declaration means unknown, not unrestricted use.

Code

Not recorded

Data

Custom AgentRewardBench Terms of Use (Publisher dataset-card Terms of Use, including third-party restrictions; no standard SPDX or unrestricted permission is asserted.)

Document

Not recorded

Weights

Not recorded

License review notes

• Data: The dataset card supplies custom Terms of Use and preserves third-party conditions. The repository has no root code license, and no dataset license is inherited from its benchmarks. Source 1

Evaluation protocol

Focus

automatic evaluator agreement, task success, side effects, repetition

Setting

offline

Observations

web trajectories, screenshots, actions

Actions / outputs

judge verdicts

Evaluation

Compare automatic judgments with expert answers on task success, side effects and repetitiveness.

Scale and counting units

• 1,302 trajectories — paper-reported expert-reviewed set source

Counts refer to the paper study; current Hub package files are not converted into trajectory totals.

Compare source-reviewed profiles

Recorded specifications

leaderboard url

https://huggingface.co/spaces/McGill-NLP/agent-reward-bench-leaderboard

Explore related topics

Scope and limitations

The public repository contains judges, evaluation utilities and annotation files; the linked Hub hosts trajectory/screenshot manifests. No root code license is declared. Dataset Terms of Use are custom and retain third-party restrictions rather than declaring an SPDX license. The paper-reported 1,302 trajectories describe its reviewed study population, not a count recomputed from current Hub files.

Related resources

• documented by: AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories

Primary evidence and verification

Source retrieval is separate from experiment reproduction. Use the original publication or publisher page when citing claims about the resource.

1. editorial primary review

   Retrieved 2026-10-17T03:59:54Z. Supports: name, kind, summary, environments, topics, scope, relations, details.

   Content fingerprint

   fb753e41c3eb19be172e9e7fccbc6bcadf44288507f82b190ca99e64144294b6
2. editorial primary review

   Retrieved 2026-10-17T03:59:54Z. Supports: identity.

   Content fingerprint

   7dfac983a6e443d5f66e1fc0fc3706ebea4c93005f03457cc0a2b13a2bd89601
3. editorial primary review

   Retrieved 2026-10-17T03:59:54Z. Supports: release_status.

   Content fingerprint

   79a1639cd4738719e51fc7f32313d63c0483914517cee96175459f63513e8c07
4. editorial primary review

   Retrieved 2026-10-17T04:03:18Z. Supports: release_status.

   Content fingerprint

   20c910fad3ad45a0c8c39b182c53632a821d3b6b546278bea6b7a78fbcb3f6cb
5. editorial primary review

   Retrieved 2026-10-17T04:03:20Z. Supports: licenses.data, summary.

   Content fingerprint

   e16adf817e3950441d70c7bb1bb3f44bd8dc65344938247f23fea4aaff9f654e
6. editorial primary review

   Retrieved 2026-10-16T06:36:54Z. Supports: details.benchmark_protocol.

   Content fingerprint

   4953a8e0f029016aef668bce98cec4d2873a7ae49c1fe33c4311fa5eb8d24089

Suggest a correction

Open a source-backed correction. Include this record ID and the original source. Read our editorial policy and coverage boundaries.