Preview: not announced. Search engines asked not to index.

Proposal pack · RO-LD06-02 v1

How faithfully do indices and policy documents relay AI evaluation results?

To keep this as a PDF or print it, use your browser's Print command (Ctrl+P, or Cmd+P on a Mac) and choose "Save as PDF". Links print with their addresses. This pack gathers what the opportunity page already holds; nothing in it is new.

Read this first. This is a research hypothesis drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-06 debate on 15 October 2026, not a finding. Not yet reviewed. Every check so far was done by AI; no person or domain expert has checked it yet. The literature screen below was done by AI web search and can miss relevant work. Version v1, permanent address https://sociolens-web.pages.dev/opportunities/ro-ld06-02/v1/.

Institutional practice unknown Master's thesis Low cost No study found

Research question

When annual indices, industry reports and government or procurement documents cite benchmark and evaluation figures, what share come from the developers themselves, what share are independently measured, and how often is the figure, its unit or its scope changed between the primary source and the relay?

Why it matters

Most readers meet evaluation results second-hand. The 2026 AI Index says it assumes company-reported results are accurate, and it presented counts of flagged items from an audit as benchmark-wide invalid rates. One case shows the risk, not its size. A systematic trace would show whether relay error and reliance on self-reports are rare slips or a structural feature of how evaluation numbers travel (P2).

The debate lists this among what we still do not know: How often do indices, procurers and regulators rely on developer-reported scores, and how often do relays distort them?

Evidence that motivates it

  1. The 2026 AI Index reports that a review found invalid-question rates in nine benchmarks “ranging from 2% on MMLU Math to 42% on GSM8K”, citing Truong et al. (2025); in that paper’s figure, 2 and 42 are the numbers of confirmed invalid questions among the 50 that a statistical screen flagged in each benchmark (“2% corresponds to one question”, so each question is 2 points on that scale), and the paper puts GSM8K’s benchmark-wide share of invalid questions at about 5%, citing Vendrow et al., while its own Platinum-based labelling marks 88 of 997 (8.8%) invalid. (LD06-SL-C01 v1.0)
    The AI Index 2026 relayed counts of invalid items among 50 flagged per benchmark as invalid-question rates of 2% to 42%.
    Cites: Truong et al. (2025), “Fantastic Bugs and Where to Find…”; Sajadieh et al. (2026), “The AI Index 2026 Annual Report”
  2. The 2026 AI Index states that the benchmark scores in its technical-performance chapter come from leaderboards, public repositories and company disclosures, including papers, blog posts and product releases, and that it “assumes that company-reported results are accurate”, while its introduction says that “independent testing does not always confirm what developers report”. (LD06-SL-C02 v1.0)
    The same index states that it assumes company-reported results are accurate.
    Cites: Sajadieh et al. (2026), “The AI Index 2026 Annual Report”
  3. The 2025 Foundation Model Transparency Index found that the average transparency score of major AI developers fell from 58 in 2024 to 40 out of 100, and that no company adequately disclosed how far its training data overlap with the evaluations it is tested on; the authors note that changed indicators and companies may contribute to the decline. (LD06-SL-C17 v1.0)
    Developer transparency, including on evaluation methods and train-test overlap, fell between the 2024 and 2025 transparency indices.
    Cites: Wan et al. (2025), “The 2025 Foundation Model…”

Claims are shown as they read on the day this pack was built; claim pages keep every earlier version.

A possible design

Draw a sample of documents from three genres (annual indices and industry reports; government strategies and regulator guidance; public procurement documents) for 2023-2026. Extract every benchmark or evaluation figure, trace it to its primary source, and code: developer-reported or independent; whether the number, its unit (share, count, points), its denominator and its scope match the source; and whether limitations stated in the source survive. Two coders with a reliability check; report error rates by genre and by source type.

Data needed

Public documents only; access to primary papers, model cards and leaderboards as of the relay date (archived copies where they have changed).

Feasibility

Master's thesis. Low; public documents and coder time. Leaderboards change over time, so the trace must use archived versions from the relay date. Results concerning named organizations should be shared with them before publication.

Has this been done?

The search found no study that does the core design (first-pass screen of 15 October 2026; not yet checked in depth).

No study was found that samples secondary documents, traces each benchmark or evaluation figure to its primary source and codes both whether it is developer-reported and whether it was relayed accurately. Related work audits academic papers for evaluating outdated models, proposes provenance labels for scores, or compares developers' self-reported scores with independent runs on one benchmark.

Where the search could not reach. Screened by web search (3 queries) and one arXiv query on the date shown; the general literature on citation and quotation errors was not searched, and HELM or Open LLM Leaderboard discrepancy studies were not looked for directly. Absence of a hit is not proof that the work does not exist.

Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.

Nearest work found

References

Sources behind the motivating claims:

BibTeX

The first entry cites this opportunity; then its nearest work; then the sources above. Nearest work and sources are recorded as reference strings, so each goes in whole as the title; add authors and year from the original before you cite it.

@misc{sociolens_ro_ld06_02_v1,
  author = {{SocioLens project}},
  title = {How faithfully do indices and policy documents relay AI evaluation results?},
  year = {2026},
  date = {2026-10-15},
  howpublished = {\url{https://sociolens-web.pages.dev/opportunities/ro-ld06-02/v1/}},
  note = {Research opportunity RO-LD06-02, version v1}
}

@misc{ro_ld06_02_nearest_1,
  title = {Gringras \& Salahshoor, "Frontier Lag: A Bibliometric Audit of Capability Misrepresentation in Academic AI Evaluation," arXiv:2605.04135 (2026)},
  url = {https://arxiv.org/abs/2605.04135},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{ro_ld06_02_nearest_2,
  title = {Ghosh, Reuel, Chim et al., "Evaluation Cards: An Interpretive Layer for AI Evaluation Reporting," arXiv:2606.09809 (2026)},
  url = {https://arxiv.org/abs/2606.09809},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{ro_ld06_02_nearest_3,
  title = {Epoch AI, "Self-reported GPQA Diamond scores vs. independent evaluations," data insight (date not confirmed)},
  url = {https://epoch.ai/data-insights/self-reported-gpqa},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only; Some details not confirmed}
}

@misc{ro_ld06_02_nearest_4,
  title = {Wang, Bianchi, Zhu, Nie, Kwon, Dhingra \& Zou, "Automated Benchmark Auditing for AI Agents and Large Language Models," arXiv:2605.26079 (2026)},
  url = {https://arxiv.org/abs/2605.26079},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{src_sl_truong_et_al_arxiv_2511_16842,
  title = {Truong, Tu, Hardy, Reuel, Tang, Burapacheep, Perera, Uwakwe, Domingue, Haber \& Koyejo, “Fantastic Bugs and Where to Find Them in AI Benchmarks,” arXiv:2511.16842, November 2025},
  url = {https://arxiv.org/abs/2511.16842},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_sl_ai_index_2026_annual_report,
  title = {Sajadieh, Fattorini, Perrault, Gil, Parli et al., “The AI Index 2026 Annual Report,” AI Index Steering Committee, Institute for Human-Centered AI, Stanford University, April 2026},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_sl_wan_et_al_fmti_2025_arxiv_2512_10169,
  title = {Wan, Klyman, Kapoor, Maslej, Longpre, Xiong, Liang \& Bommasani, “The 2025 Foundation Model Transparency Index,” arXiv:2512.10169, December 2025},
  url = {https://arxiv.org/abs/2512.10169},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

Back to the opportunity · All research opportunities