An audit of Chatbot Arena’s data from January 2024 to April 2025 found undisclosed private testing (27 private Meta variants before the Llama 4 release), data skewed towards proprietary providers (Google and OpenAI about 20% each), and 205 of 243 public models silently deprecated, and concluded that rankings partly reflect adaptation to the Arena rather than general model quality.

SocioLens project,

Claim LD06-SL-C15 (v1.0)

Provenance and status

Claim type: Estimate
Publication status: Eligible for publication, with qualifiers: source position
Verification state: Checked against the primary source; the source is a preprint. Who checked this claim is not recorded for it; claim checks so far were AI-led, and no person has checked claims against their sources.
Does not license: That any provider broke the Arena’s rules; the size of the private-testing effect on public scores, which the platform disputes
Limitations: Authors compete on the Arena, and Cohere’s own enquiry prompted the work; the private-testing effect rests partly on simulation; the platform’s response disputes several figures (LD06-SL-C16)
Source (as cited): Singh, Nan, Wang, D’Souza, Kapoor, Üstün, Koyejo, Deng, Longpre, Smith, Ermis, Fadaee & Hooker, “The Leaderboard Illusion,” arXiv:2504.20879 (v2), May 2025
Source type: Preprint (recorded as “Preprint”)
Institutional position: AI developer-led (Cohere, an Arena participant) with academic co-authors
Maturity of the evidence: Contested evidence
Recorded date: 15 October 2026
Currentness / correction: This is the current version. No correction or concern is recorded on its sources.

Sources this version cites

Singh, Nan, Wang, D’Souza, Kapoor, Üstün, Koyejo, Deng, Longpre, Smith, Ermis, Fadaee & Hooker, “The Leaderboard Illusion,” arXiv:2504.20879 (v2), May 2025 (Preprint)
Read the original

Version

This is the current version, v1.0. Earlier versions stay available, and their records never change.

Versions: v1.0 · current

Raw record: every recorded field, with its coded values

All 25 canonical Claim fields, verbatim, with the coded values derived from them. The plain-language labels above are read from these.

Claim type (coded): ESTIMATE (as printed: “ESTIMATE”)
Publication status (coded): ELIGIBLE (SOURCE_POSITION) (as printed: “ELIGIBLE WITH SOURCE-POSITION QUALIFIER”)
Verification state (coded): CONTENT_VERIFIED_PRIMARY (PREPRINT)
Source type (coded): PREPRINT
Maturity (coded): CONTESTED

#	Canonical field	Canonical value (verbatim)	Normalized (Claim Value Vocabulary v1.0.2)
1	CLAIM ID	LD06-SL-C15	—
2	PUBLIC-FACING CLAIM	An audit of Chatbot Arena’s data from January 2024 to April 2025 found undisclosed private testing (27 private Meta variants before the Llama 4 release), data skewed towards proprietary providers (Google and OpenAI about 20% each), and 205 of 243 public models silently deprecated, and concluded that rankings partly reflect adaptation to the Arena rather than general model quality.	—
3	CLAIM TYPE	ESTIMATE	base_types: ESTIMATE
4	RESEARCH STRAND	B — Leaderboards, judges and disclosure	—
5	MATURITY	CONTESTED	states: CONTESTED
6	SOURCE	Singh, Nan, Wang, D’Souza, Kapoor, Üstün, Koyejo, Deng, Longpre, Smith, Ermis, Fadaee & Hooker, “The Leaderboard Illusion,” arXiv:2504.20879 (v2), May 2025	source_ids: SRC-SL-SINGH-ET-AL-LEADERBOARD-ILLUSION-ARXIV-2504-20879
7	SOURCE TYPE	Preprint	classes: PREPRINT
8	INSTITUTIONAL POSITION	AI developer-led (Cohere, an Arena participant) with academic co-authors	—
9	SCOPE	Chatbot Arena, January 2024 to April 2025	—
10	DESIGN	Analysis of battle data, scraped records and simulations, plus the authors’ own private-variant experiment	—
11	SAMPLE / DATASET	2 million battles, 42 providers, 243 models	—
12	MODEL / TOOL / VERSION	Models ranked on the Arena	—
13	COMPARATOR	Simulated leaderboards; training with and without Arena data	—
14	MAIN RESULT	27 private Meta variants; Google 19.2% and OpenAI 20.4% of Arena data; 41 fully open-source models received an estimated 8.9% of data (v2); 205 of 243 models silently deprecated; in the authors’ own fine-tuning experiment, adding Arena data raised win rates on ArenaHard from 23.5% to 49.9%	—
15	INFERENTIAL TARGET	Whether a preference leaderboard ranks general model quality	—
16	DOES NOT LICENSE	That any provider broke the Arena’s rules; the size of the private-testing effect on public scores, which the platform disputes	—
17	VERIFICATION STATE	CONTENT VERIFIED PREPRINT	state: CONTENT_VERIFIED_PRIMARY · qualifiers: PREPRINT · verified_by: UNKNOWN
18	LIMITATIONS	Authors compete on the Arena, and Cohere’s own enquiry prompted the work; the private-testing effect rests partly on simulation; the platform’s response disputes several figures (LD06-SL-C16)	—
19	CONTRADICTORY / SUPPORTING SOURCES	Disputed in part by LD06-SL-C16	—
20	GEOGRAPHY / LANGUAGE	Global platform / English	—
21	DISTRIBUTIONAL LEDGER	Open-weight and open-source models received less data and were deprecated more often	—
22	R01 / FIELD-MAP LINKS	LD-06, D4b	—
23	WHAT WOULD CHANGE THIS CLAIM?	Platform data showing that private testing does not move public ranks	—
24	PUBLICATION STATUS	ELIGIBLE WITH SOURCE-POSITION QUALIFIER	state: ELIGIBLE · qualifiers: SOURCE_POSITION
25	RECORDED DATE	2026-10-15	—

Field 22 references resolved against Field Map v0.2: LD-06 (LIVING_DEBATE); D4b (SUB_AREA)

The sources as recorded, with their record IDs and coded source types:

Pinned Sources (field 6, version-specific)

“Singh, Nan, Wang, D’Souza, Kapoor, Üstün, Koyejo, Deng, Longpre, Smith, Ermis, Fadaee & Hooker, “The Leaderboard Illusion,” arXiv:2504.20879 (v2), May 2025” → Singh et al. (2025), “The Leaderboard Illusion” (SRC-SL-SINGH-ET-AL-LEADERBOARD-ILLUSION-ARXIV-2504-20879--v1.0) — Singh, Nan, Wang, D’Souza, Kapoor, Üstün, Koyejo, Deng, Longpre, Smith, Ermis, Fadaee & Hooker, “The Leaderboard Illusion,” arXiv:2504.20879 (v2), May 2025; class PREPRINT (explicit public provenance pin)

Cited by Living Debate versions

The evaluation apparatus: Can benchmarks, leaderboards and audits carry the weight placed on them? (v0.1) LD-06

Cite this page

SocioLens project. (2026, October 15). An audit of Chatbot Arena’s data from January 2024 to April 2025 found undisclosed private testing (27 private Meta variants before the Llama 4 release), data skewed towards proprietary providers (Google and OpenAI about 20% each), and 205 of 243 public models silently deprecated, and concluded that rankings partly reflect adaptation to the Arena rather than general model quality [Claim LD06-SL-C15, version v1.0]. SocioLens. https://sociolens-web.pages.dev/claims/ld06-sl-c15/v1.0/

@misc{sociolens_ld06_sl_c15_v1_0,
  author = {{SocioLens project}},
  title = {An audit of Chatbot Arena’s data from January 2024 to April 2025 found undisclosed private testing (27 private Meta variants before the Llama 4 release), data skewed towards proprietary providers (Google and OpenAI about 20\% each), and 205 of 243 public models silently deprecated, and concluded that rankings partly reflect adaptation to the Arena rather than general model quality.},
  year = {2026},
  date = {2026-10-15},
  howpublished = {\url{https://sociolens-web.pages.dev/claims/ld06-sl-c15/v1.0/}},
  note = {Claim LD06-SL-C15, version v1.0}
}

The link points to this version, whose record never changes. For a finding, cite the original study as well: SocioLens summarizes research and does not replace it.

Research-of-record provenance

Materialized by sociolens-materializer 0.6.0 at 2026-09-28T20:00:00.000Z; normalization contract SocioLens — Claim Value Vocabulary v1.0.2 — Pre-Spike Materialization Contract (v1.0.2). Content seal SHA-256 533127625564ce3cf2885f4120d0fac508579accec20925a614c903ec855e9ff.

Repository-authored inputs (not Drive exports)

SocioLens claim — LD06-SL-C15 v1.0 (v1.0) — written in the SocioLens repository by Project Lead (Claude, AI) on 2026-10-15; no Drive identity exists. Authority: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts. Block “ENTRY 15 — LD06-SL-C15” SHA-256 9eb78693321ac2ee3347b541e1139f40077e79112d77755951ff2e19def963e8; fixture fixtures/authored/claims/LD06-SL-C15--v1.0.

Builder decisions (not Drive data)

VERSION = v1.0 — basis: Claim version taken from the repository-authored claim fixture version (SocioLens claim — LD06-SL-C15 v1.0); the first version of a new Claim of the LD-06 debate
CURRENT_POINTER = true — basis: repository-authored claim "SocioLens claim — LD06-SL-C15 v1.0" (Project Lead (Claude, AI), 2026-10-15); basis: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts
FIELD_6_SOURCE_REFERENCE = SRC-SL-SINGH-ET-AL-LEADERBOARD-ILLUSION-ARXIV-2504-20879--v1.0 — basis: field 6 of ENTRY 15 — LD06-SL-C15 cites "Singh, Nan, Wang, D’Souza, Kapoor, Üstün, Koyejo, Deng, Longpre, Smith, Ermis, Fadaee & Hooker, “The Leaderboard Illusion,” arXiv:2504.20879 (v2), May 2025"; the only field-6 segment of this ClaimVersion's repository-authored ledger block.
SOURCE_PROVENANCE_PIN = SRC-SL-SINGH-ET-AL-LEADERBOARD-ILLUSION-ARXIV-2504-20879--v1.0 — basis: explicit version-pinned public provenance of this ClaimVersion (Amendment v1.2 §6): the pinned field-6 Source record "Singh, Nan, Wang, D’Souza, Kapoor, Üstün, Koyejo, Deng, Longpre, Smith, Ermis, Fadaee & Hooker, “The Leaderboard Illusion,” arXiv:2504.20879 (v2), May 2025".