An audit of agentic benchmarks found flaws in task setup or scoring, for example insufficient test cases in SWE-bench Verified and empty responses counted as successes in TAU-bench, that can under- or overestimate agents’ performance by up to 100% in relative terms.

SocioLens project,

Claim LD06-SL-C08 (v1.0)

Provenance and status

Claim type: Estimate
Publication status: Eligible for publication, with qualifiers: preprint
Verification state: Checked against the primary source; the source is a preprint. Who checked this claim is not recorded for it; claim checks so far were AI-led, and no person has checked claims against their sources.
Does not license: A correction factor for any published agent score
Limitations: Read in full text (arXiv v5); venue not checked; the audit is by benchmark builders applying their own checklist
Source (as cited): Zhu, Jin, Pruksachatkun, Zhang, Liu, Cui et al., “Establishing Best Practices for Building Rigorous Agentic Benchmarks,” arXiv:2507.02825 (v5), 2025
Source type: Preprint (recorded as “Preprint”)
Institutional position: Independent academic and nonprofit (UIUC-led, with Stanford, Berkeley, the UK AI Security Institute and others); CVE-Bench, the benchmark the authors improve, comes from the lead authors’ own group
Maturity of the evidence: Emerging evidence
Recorded date: 15 October 2026
Currentness / correction: This is the current version. No correction or concern is recorded on its sources.

Sources this version cites

Zhu, Jin, Pruksachatkun, Zhang, Liu, Cui et al., “Establishing Best Practices for Building Rigorous Agentic Benchmarks,” arXiv:2507.02825 (v5), 2025 (Preprint)
Read the original

Version

This page shows version v1.0. Its record never changes, even when newer evidence arrives; the site may add a note to the page or withhold a figure not cleared for public use.

This pinned version is also the current version.

Versions: v1.0 · current

Raw record: every recorded field, with its coded values

All 25 canonical Claim fields, verbatim, with the coded values derived from them. The plain-language labels above are read from these.

Claim type (coded): ESTIMATE (as printed: “ESTIMATE”)
Publication status (coded): ELIGIBLE (PREPRINT) (as printed: “ELIGIBLE WITH PREPRINT QUALIFIER”)
Verification state (coded): CONTENT_VERIFIED_PRIMARY (PREPRINT)
Source type (coded): PREPRINT
Maturity (coded): EMERGING

#	Canonical field	Canonical value (verbatim)	Normalized (Claim Value Vocabulary v1.0.2)
1	CLAIM ID	LD06-SL-C08	—
2	PUBLIC-FACING CLAIM	An audit of agentic benchmarks found flaws in task setup or scoring, for example insufficient test cases in SWE-bench Verified and empty responses counted as successes in TAU-bench, that can under- or overestimate agents’ performance by up to 100% in relative terms.	—
3	CLAIM TYPE	ESTIMATE	base_types: ESTIMATE
4	RESEARCH STRAND	A — Benchmark validity and measurement	—
5	MATURITY	EMERGING	states: EMERGING
6	SOURCE	Zhu, Jin, Pruksachatkun, Zhang, Liu, Cui et al., “Establishing Best Practices for Building Rigorous Agentic Benchmarks,” arXiv:2507.02825 (v5), 2025	source_ids: SRC-SL-ZHU-ET-AL-AGENTIC-BENCHMARK-CHECKLIST-ARXIV-2507-02825
7	SOURCE TYPE	Preprint	classes: PREPRINT
8	INSTITUTIONAL POSITION	Independent academic and nonprofit (UIUC-led, with Stanford, Berkeley, the UK AI Security Institute and others); CVE-Bench, the benchmark the authors improve, comes from the lead authors’ own group	—
9	SCOPE	Benchmarks that score AI agents on multi-step tasks	—
10	DESIGN	Audit of benchmark designs; checklist synthesized and applied	—
11	SAMPLE / DATASET	Ten widely used agentic benchmarks	—
12	MODEL / TOOL / VERSION	Agents evaluated on the audited benchmarks	—
13	COMPARATOR	Corrected task setups and reward designs	—
14	MAIN RESULT	Relative under- or overestimation of up to 100%; applying the checklist to CVE-Bench reduced overestimation by 33%	—
15	INFERENTIAL TARGET	Whether agent benchmark scores measure task success	—
16	DOES NOT LICENSE	A correction factor for any published agent score	—
17	VERIFICATION STATE	CONTENT VERIFIED PREPRINT	state: CONTENT_VERIFIED_PRIMARY · qualifiers: PREPRINT · verified_by: UNKNOWN
18	LIMITATIONS	Read in full text (arXiv v5); venue not checked; the audit is by benchmark builders applying their own checklist	—
19	CONTRADICTORY / SUPPORTING SOURCES	Consistent with maintainer review of SWE-bench patches: LD06-SL-C12	—
20	GEOGRAPHY / LANGUAGE	Not checked / English	—
21	DISTRIBUTIONAL LEDGER	Not analysed in this claim	—
22	R01 / FIELD-MAP LINKS	LD-06, D4b	—
23	WHAT WOULD CHANGE THIS CLAIM?	Independent re-scoring of published agent results under corrected designs	—
24	PUBLICATION STATUS	ELIGIBLE WITH PREPRINT QUALIFIER	state: ELIGIBLE · qualifiers: PREPRINT
25	RECORDED DATE	2026-10-15	—

Field 22 references resolved against Field Map v0.2: LD-06 (LIVING_DEBATE); D4b (SUB_AREA)

The sources as recorded, with their record IDs and coded source types:

Pinned Sources (field 6, version-specific)

“Zhu, Jin, Pruksachatkun, Zhang, Liu, Cui et al., “Establishing Best Practices for Building Rigorous Agentic Benchmarks,” arXiv:2507.02825 (v5), 2025” → Zhu et al. (2025), “Establishing Best Practices for…” (SRC-SL-ZHU-ET-AL-AGENTIC-BENCHMARK-CHECKLIST-ARXIV-2507-02825--v1.0) — Zhu, Jin, Pruksachatkun, Zhang, Liu, Cui et al., “Establishing Best Practices for Building Rigorous Agentic Benchmarks,” arXiv:2507.02825 (v5), 2025; class PREPRINT (explicit public provenance pin)

Cited by Living Debate versions

The evaluation apparatus: Can benchmarks, leaderboards and audits carry the weight placed on them? (v0.1) LD-06

Cite this page

SocioLens project. (2026, October 15). An audit of agentic benchmarks found flaws in task setup or scoring, for example insufficient test cases in SWE-bench Verified and empty responses counted as successes in TAU-bench, that can under- or overestimate agents’ performance by up to 100% in relative terms [Claim LD06-SL-C08, version v1.0]. SocioLens. https://sociolens-web.pages.dev/claims/ld06-sl-c08/v1.0/

@misc{sociolens_ld06_sl_c08_v1_0,
  author = {{SocioLens project}},
  title = {An audit of agentic benchmarks found flaws in task setup or scoring, for example insufficient test cases in SWE-bench Verified and empty responses counted as successes in TAU-bench, that can under- or overestimate agents’ performance by up to 100\% in relative terms.},
  year = {2026},
  date = {2026-10-15},
  howpublished = {\url{https://sociolens-web.pages.dev/claims/ld06-sl-c08/v1.0/}},
  note = {Claim LD06-SL-C08, version v1.0}
}

The link points to this version, whose record never changes. For a finding, cite the original study as well: SocioLens summarizes research and does not replace it.

Research-of-record provenance

Materialized by sociolens-materializer 0.6.0 at 2026-09-28T20:00:00.000Z; normalization contract SocioLens — Claim Value Vocabulary v1.0.2 — Pre-Spike Materialization Contract (v1.0.2). Content seal SHA-256 29b90ab9c8a4ef851d695405d378a42ac080f1aea67ad4b14f7d59b7873e1a82.

Repository-authored inputs (not Drive exports)

SocioLens claim — LD06-SL-C08 v1.0 (v1.0) — written in the SocioLens repository by Project Lead (Claude, AI) on 2026-10-15; no Drive identity exists. Authority: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts. Block “ENTRY 08 — LD06-SL-C08” SHA-256 053ce3805c89335874ed64ce9a7ed465a2db9be1553b754672eec6f24b5d6efb; fixture fixtures/authored/claims/LD06-SL-C08--v1.0.

Builder decisions (not Drive data)

VERSION = v1.0 — basis: Claim version taken from the repository-authored claim fixture version (SocioLens claim — LD06-SL-C08 v1.0); the first version of a new Claim of the LD-06 debate
CURRENT_POINTER = true — basis: repository-authored claim "SocioLens claim — LD06-SL-C08 v1.0" (Project Lead (Claude, AI), 2026-10-15); basis: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts
FIELD_6_SOURCE_REFERENCE = SRC-SL-ZHU-ET-AL-AGENTIC-BENCHMARK-CHECKLIST-ARXIV-2507-02825--v1.0 — basis: field 6 of ENTRY 08 — LD06-SL-C08 cites "Zhu, Jin, Pruksachatkun, Zhang, Liu, Cui et al., “Establishing Best Practices for Building Rigorous Agentic Benchmarks,” arXiv:2507.02825 (v5), 2025"; the only field-6 segment of this ClaimVersion's repository-authored ledger block.
SOURCE_PROVENANCE_PIN = SRC-SL-ZHU-ET-AL-AGENTIC-BENCHMARK-CHECKLIST-ARXIV-2507-02825--v1.0 — basis: explicit version-pinned public provenance of this ClaimVersion (Amendment v1.2 §6): the pinned field-6 Source record "Zhu, Jin, Pruksachatkun, Zhang, Liu, Cui et al., “Establishing Best Practices for Building Rigorous Agentic Benchmarks,” arXiv:2507.02825 (v5), 2025".