In interviews with 19 people who use or decided against using AI benchmarks, academics generally saw public benchmarks as suitable for tracking research progress, while in product and policy work benchmarks, even internally built ones, were often found inadequate for substantive decisions.

SocioLens project,

Claim LD06-SL-C13 (v1.0)

Provenance and status

Claim type: Estimate
Publication status: Eligible for publication, with a qualifier
Verification state: Checked against the abstract of the primary source only. Who checked this claim is not recorded for it; claim checks so far were AI-led, and no person has checked claims against their sources.
Does not license: A prevalence estimate of benchmark use or distrust
Limitations: Read in abstract and first page only; small qualitative sample
Source (as cited): Hardy, Reuel, Jafari Meimandi, Soder, Griffith, Asmar, Koyejo, Bernstein & Kochenderfer, “More than Marketing? On the Information Value of AI Benchmarks for Practitioners,” arXiv:2412.05520, December 2024
Source type: Preprint (recorded as “Preprint”)
Institutional position: Independent academic (Stanford, with one author at the policy think tank interface)
Maturity of the evidence: Emerging evidence
Recorded date: 15 October 2026
Currentness / correction: This is the current version. No correction or concern is recorded on its sources.

Sources this version cites

Hardy, Reuel, Jafari Meimandi, Soder, Griffith, Asmar, Koyejo, Bernstein & Kochenderfer, “More than Marketing? On the Information Value of AI Benchmarks for Practitioners,” arXiv:2412.05520, December 2024 (Preprint)
Read the original

Version

This page shows version v1.0. Its record never changes, even when newer evidence arrives; the site may add a note to the page or withhold a figure not cleared for public use.

This pinned version is also the current version.

Versions: v1.0 · current

Raw record: every recorded field, with its coded values

All 25 canonical Claim fields, verbatim, with the coded values derived from them. The plain-language labels above are read from these.

Claim type (coded): ESTIMATE (as printed: “ESTIMATE”)
Publication status (coded): ELIGIBLE (QUALIFIED) (as printed: “ELIGIBLE WITH QUALIFIER”)
Verification state (coded): CONTENT_VERIFIED_PRIMARY (ABSTRACT_ONLY)
Source type (coded): PREPRINT
Maturity (coded): EMERGING

#	Canonical field	Canonical value (verbatim)	Normalized (Claim Value Vocabulary v1.0.2)
1	CLAIM ID	LD06-SL-C13	—
2	PUBLIC-FACING CLAIM	In interviews with 19 people who use or decided against using AI benchmarks, academics generally saw public benchmarks as suitable for tracking research progress, while in product and policy work benchmarks, even internally built ones, were often found inadequate for substantive decisions.	—
3	CLAIM TYPE	ESTIMATE	base_types: ESTIMATE
4	RESEARCH STRAND	A — Benchmark validity and measurement	—
5	MATURITY	EMERGING	states: EMERGING
6	SOURCE	Hardy, Reuel, Jafari Meimandi, Soder, Griffith, Asmar, Koyejo, Bernstein & Kochenderfer, “More than Marketing? On the Information Value of AI Benchmarks for Practitioners,” arXiv:2412.05520, December 2024	source_ids: SRC-SL-HARDY-ET-AL-MORE-THAN-MARKETING-ARXIV-2412-05520
7	SOURCE TYPE	Preprint	classes: PREPRINT
8	INSTITUTIONAL POSITION	Independent academic (Stanford, with one author at the policy think tank interface)	—
9	SCOPE	Practitioners in academia, product development and policy	—
10	DESIGN	Qualitative interview study	—
11	SAMPLE / DATASET	19 interviewees	—
12	MODEL / TOOL / VERSION	Not applicable	—
13	COMPARATOR	Between academic, product and policy settings	—
14	MAIN RESULT	Benchmarks used as a signal of relative performance; in product and policy, often inadequate for decisions because goals were neither well defined nor reflective of real-world use	—
15	INFERENTIAL TARGET	Whether benchmark scores carry over to the decisions people make with them	—
16	DOES NOT LICENSE	A prevalence estimate of benchmark use or distrust	—
17	VERIFICATION STATE	CONTENT VERIFIED ABSTRACT	state: CONTENT_VERIFIED_PRIMARY · qualifiers: ABSTRACT_ONLY · verified_by: UNKNOWN
18	LIMITATIONS	Read in abstract and first page only; small qualitative sample	—
19	CONTRADICTORY / SUPPORTING SOURCES	Compare the evaluation gap named in LD06-SL-C14	—
20	GEOGRAPHY / LANGUAGE	Not checked / English	—
21	DISTRIBUTIONAL LEDGER	Not analysed in this claim	—
22	R01 / FIELD-MAP LINKS	LD-06, D4b	—
23	WHAT WOULD CHANGE THIS CLAIM?	Larger surveys of procurement and policy users	—
24	PUBLICATION STATUS	ELIGIBLE WITH QUALIFIER	state: ELIGIBLE · qualifiers: QUALIFIED
25	RECORDED DATE	2026-10-15	—

Field 22 references resolved against Field Map v0.2: LD-06 (LIVING_DEBATE); D4b (SUB_AREA)

The sources as recorded, with their record IDs and coded source types:

Pinned Sources (field 6, version-specific)

“Hardy, Reuel, Jafari Meimandi, Soder, Griffith, Asmar, Koyejo, Bernstein & Kochenderfer, “More than Marketing? On the Information Value of AI Benchmarks for Practitioners,” arXiv:2412.05520, December 2024” → Hardy et al. (2024), “More than Marketing? On the…” (SRC-SL-HARDY-ET-AL-MORE-THAN-MARKETING-ARXIV-2412-05520--v1.0) — Hardy, Reuel, Jafari Meimandi, Soder, Griffith, Asmar, Koyejo, Bernstein & Kochenderfer, “More than Marketing? On the Information Value of AI Benchmarks for Practitioners,” arXiv:2412.05520, December 2024; class PREPRINT (explicit public provenance pin)

Cited by Living Debate versions

The evaluation apparatus: Can benchmarks, leaderboards and audits carry the weight placed on them? (v0.1) LD-06

Cite this page

SocioLens project. (2026, October 15). In interviews with 19 people who use or decided against using AI benchmarks, academics generally saw public benchmarks as suitable for tracking research progress, while in product and policy work benchmarks, even internally built ones, were often found inadequate for substantive decisions [Claim LD06-SL-C13, version v1.0]. SocioLens. https://sociolens-web.pages.dev/claims/ld06-sl-c13/v1.0/

@misc{sociolens_ld06_sl_c13_v1_0,
  author = {{SocioLens project}},
  title = {In interviews with 19 people who use or decided against using AI benchmarks, academics generally saw public benchmarks as suitable for tracking research progress, while in product and policy work benchmarks, even internally built ones, were often found inadequate for substantive decisions.},
  year = {2026},
  date = {2026-10-15},
  howpublished = {\url{https://sociolens-web.pages.dev/claims/ld06-sl-c13/v1.0/}},
  note = {Claim LD06-SL-C13, version v1.0}
}

The link points to this version, whose record never changes. For a finding, cite the original study as well: SocioLens summarizes research and does not replace it.

Research-of-record provenance

Materialized by sociolens-materializer 0.6.0 at 2026-09-28T20:00:00.000Z; normalization contract SocioLens — Claim Value Vocabulary v1.0.2 — Pre-Spike Materialization Contract (v1.0.2). Content seal SHA-256 f8c385164c90fb01c5c8a212a3b9748679726d59d3954d21311e738703d0eb8a.

Repository-authored inputs (not Drive exports)

SocioLens claim — LD06-SL-C13 v1.0 (v1.0) — written in the SocioLens repository by Project Lead (Claude, AI) on 2026-10-15; no Drive identity exists. Authority: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts. Block “ENTRY 13 — LD06-SL-C13” SHA-256 7e6712e470f9064cc597511763903c97fdc20f4f4dfe4a79faf41cda8f7c6aa2; fixture fixtures/authored/claims/LD06-SL-C13--v1.0.

Builder decisions (not Drive data)

VERSION = v1.0 — basis: Claim version taken from the repository-authored claim fixture version (SocioLens claim — LD06-SL-C13 v1.0); the first version of a new Claim of the LD-06 debate
CURRENT_POINTER = true — basis: repository-authored claim "SocioLens claim — LD06-SL-C13 v1.0" (Project Lead (Claude, AI), 2026-10-15); basis: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts
FIELD_6_SOURCE_REFERENCE = SRC-SL-HARDY-ET-AL-MORE-THAN-MARKETING-ARXIV-2412-05520--v1.0 — basis: field 6 of ENTRY 13 — LD06-SL-C13 cites "Hardy, Reuel, Jafari Meimandi, Soder, Griffith, Asmar, Koyejo, Bernstein & Kochenderfer, “More than Marketing? On the Information Value of AI Benchmarks for Practitioners,” arXiv:2412.05520, December 2024"; the only field-6 segment of this ClaimVersion's repository-authored ledger block.
SOURCE_PROVENANCE_PIN = SRC-SL-HARDY-ET-AL-MORE-THAN-MARKETING-ARXIV-2412-05520--v1.0 — basis: explicit version-pinned public provenance of this ClaimVersion (Amendment v1.2 §6): the pinned field-6 Source record "Hardy, Reuel, Jafari Meimandi, Soder, Griffith, Asmar, Koyejo, Bernstein & Kochenderfer, “More than Marketing? On the Information Value of AI Benchmarks for Practitioners,” arXiv:2412.05520, December 2024".