Research opportunity · The evaluation apparatus: Can benchmarks, leaderboards and audits carry the weight placed on them?
How faithfully do indices and policy documents relay AI evaluation results?
In short: has this been done?
Related work: No study found. A first-pass search found no study that does the core design. It has not been checked in depth, so a fuller search could still find one.
How far checked: First-pass screen only. A first-pass AI web search on 15 October 2026. No deeper check yet. Every search and check so far was done by AI; no person or domain expert has checked them yet.
The searches, the nearest work found and what the search could not reach
This page shows RO-LD06-02 version v1, whose record never changes. It is the current version; if the opportunity is revised, the newer version appears at its main page.
Question
When annual indices, industry reports and government or procurement documents cite benchmark and evaluation figures, what share come from the developers themselves, what share are independently measured, and how often is the figure, its unit or its scope changed between the primary source and the relay?
Gap type: Institutional practice unknownMaster's thesisLow costRelated work: No study found
Why it matters
Most readers meet evaluation results second-hand. The 2026 AI Index says it assumes company-reported results are accurate, and it presented counts of flagged items from an audit as benchmark-wide invalid rates. One case shows the risk, not its size. A systematic trace would show whether relay error and reliance on self-reports are rare slips or a structural feature of how evaluation numbers travel (P2).
The debate lists this among what we still do not know: How often do indices, procurers and regulators rely on developer-reported scores, and how often do relays distort them?
Evidence that motivates it
- The 2026 AI Index reports that a review found invalid-question rates in nine benchmarks “ranging from 2% on MMLU Math to 42% on GSM8K”, citing Truong et al. (2025); in that paper’s figure, 2 and 42 are the numbers of confirmed invalid questions among the 50 that a statistical screen flagged in each benchmark (“2% corresponds to one question”, so each question is 2 points on that scale), and the paper puts GSM8K’s benchmark-wide share of invalid questions at about 5%, citing Vendrow et al., while its own Platinum-based labelling marks 88 of 997 (8.8%) invalid. (LD06-SL-C01 v1.0)
How this study uses it: The AI Index 2026 relayed counts of invalid items among 50 flagged per benchmark as invalid-question rates of 2% to 42%. - The 2026 AI Index states that the benchmark scores in its technical-performance chapter come from leaderboards, public repositories and company disclosures, including papers, blog posts and product releases, and that it “assumes that company-reported results are accurate”, while its introduction says that “independent testing does not always confirm what developers report”. (LD06-SL-C02 v1.0)
How this study uses it: The same index states that it assumes company-reported results are accurate. - The 2025 Foundation Model Transparency Index found that the average transparency score of major AI developers fell from 58 in 2024 to 40 out of 100, and that no company adequately disclosed how far its training data overlap with the evaluations it is tested on; the authors note that changed indicators and companies may contribute to the decline. (LD06-SL-C17 v1.0)
How this study uses it: Developer transparency, including on evaluation methods and train-test overlap, fell between the 2024 and 2025 transparency indices.
Each claim is shown as it read when this version was published.
A possible design
Draw a sample of documents from three genres (annual indices and industry reports; government strategies and regulator guidance; public procurement documents) for 2023-2026. Extract every benchmark or evaluation figure, trace it to its primary source, and code: developer-reported or independent; whether the number, its unit (share, count, points), its denominator and its scope match the source; and whether limitations stated in the source survive. Two coders with a reliability check; report error rates by genre and by source type.
Data needed
Public documents only; access to primary papers, model cards and leaderboards as of the relay date (archived copies where they have changed).
Feasibility
Master's thesis. Low; public documents and coder time. Leaderboards change over time, so the trace must use archived versions from the relay date. Results concerning named organizations should be shared with them before publication.
Has this been done?
The search found no study that does the core design (first-pass screen of 15 October 2026; not yet checked in depth).
No study was found that samples secondary documents, traces each benchmark or evaluation figure to its primary source and codes both whether it is developer-reported and whether it was relayed accurately. Related work audits academic papers for evaluating outdated models, proposes provenance labels for scores, or compares developers' self-reported scores with independent runs on one benchmark.
How strong this evidence is
- Who searched and checked: no checker is recorded beyond the first-pass screen, which was part of the AI drafting (Claude (SocioLens Project Lead), AI-drafted from the LD-06 debate). Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: 4 of the 4 nearest works below were judged from an abstract, summary or search listing only, and 1 has details that could not be confirmed; each is marked. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search (3 queries) and one arXiv query on the date shown; the general literature on citation and quotation errors was not searched, and HELM or Open LLM Leaderboard discrepancy studies were not looked for directly. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- Gringras & Salahshoor, "Frontier Lag: A Bibliometric Audit of Capability Misrepresentation in Academic AI Evaluation," arXiv:2605.04135 (2026) Judged from an abstract, summary or search listing only
Abstract read. Measures how far evaluated models lag the frontier in academic papers; the claim that results spread to policy was seen in search results only. - Ghosh, Reuel, Chim et al., "Evaluation Cards: An Interpretive Layer for AI Evaluation Reporting," arXiv:2606.09809 (2026) Judged from an abstract, summary or search listing only
Abstract read. Labels scores as first-party, third-party or collaborative; reporting infrastructure, not a relay audit. - Epoch AI, "Self-reported GPQA Diamond scores vs. independent evaluations," data insight (date not confirmed) Judged from an abstract, summary or search listing onlySome details not confirmed
Judged from search listings; not read directly (the page was blocked). Developer-versus-independent comparison for one benchmark. - Wang, Bianchi, Zhu, Nie, Kwon, Dhingra & Zou, "Automated Benchmark Auditing for AI Agents and Large Language Models," arXiv:2605.26079 (2026) Judged from an abstract, summary or search listing only
Abstract read. Audits 168 benchmarks for item issues; benchmark validity, not relay accuracy.
Search queries used
self-reported benchmark results discrepancy independent evaluation model developers reproduce scoresaudit of AI Index report benchmark figures provenance developer-reported scores policy documents citation accuracypolicy documents misreport AI benchmark results citation distortion relayed scores government reports studyarXiv: abs:"self-reported" AND abs:benchmark AND abs:scores
Cite this page
SocioLens project. (2026, October 15). How faithfully do indices and policy documents relay AI evaluation results? [Research opportunity RO-LD06-02, version v1]. SocioLens. https://sociolens-web.pages.dev/opportunities/ro-ld06-02/v1/
@misc{sociolens_ro_ld06_02_v1,
author = {{SocioLens project}},
title = {How faithfully do indices and policy documents relay AI evaluation results?},
year = {2026},
date = {2026-10-15},
howpublished = {\url{https://sociolens-web.pages.dev/opportunities/ro-ld06-02/v1/}},
note = {Research opportunity RO-LD06-02, version v1}
}The link points to this version, whose record never changes. For a finding, cite the original study as well: SocioLens summarizes research and does not replace it.
Provenance
RO-LD06-02 v1. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-06 debate on 2026-10-15. Not yet reviewed. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.