Responding to “The Leaderboard Illusion”, LMArena said providers may test several private variants, promised to mark scores provisional until 2,000 fresh votes when more than 10 variants are tested, and disputed the paper’s figures, estimating the gain from private testing at about 11 Elo points; the paper’s 8.8% open-source figure that it disputed was a share of data, not of models.

SocioLens project,

Claim LD06-SL-C16 (v1.0)

Provenance and status

Claim type: Fact and Interpretation
Publication status: Eligible for publication, with qualifiers: source position
Verification state: Checked against the primary source. Who checked this claim is not recorded for it; claim checks so far were AI-led, and no person has checked claims against their sources.
Does not license: A finding on which side’s estimate of the private-testing effect is right
Limitations: An interested party’s account; the comparison of the two texts is SocioLens’s reading; neither estimate was re-derived
Source (as cited): LMArena, “Our Response to ‘The Leaderboard Illusion’ Writeup,” Arena blog, 9 May 2025 (updated 13 June 2025)
Source type: Vendor or company document (recorded as “Platform operator blog post”)
Institutional position: Platform operator (LMArena), the subject of the criticism
Maturity of the evidence: Contested evidence
Recorded date: 15 October 2026
Currentness / correction: This is the current version. No correction or concern is recorded on its sources.

Sources this version cites

LMArena, “Our Response to ‘The Leaderboard Illusion’ Writeup,” Arena blog, 9 May 2025 (updated 13 June 2025) (Vendor or company document)
Read the original

Version

This is the current version, v1.0. Earlier versions stay available, and their records never change.

Versions: v1.0 · current

Raw record: every recorded field, with its coded values

All 25 canonical Claim fields, verbatim, with the coded values derived from them. The plain-language labels above are read from these.

Claim type (coded): FACT + INTERPRETATION (as printed: “FACT + INTERPRETATION”)
Publication status (coded): ELIGIBLE (SOURCE_POSITION) (as printed: “ELIGIBLE WITH SOURCE-POSITION QUALIFIER”)
Verification state (coded): CONTENT_VERIFIED_PRIMARY
Source type (coded): VENDOR_PRIMARY
Maturity (coded): CONTESTED

#	Canonical field	Canonical value (verbatim)	Normalized (Claim Value Vocabulary v1.0.2)
1	CLAIM ID	LD06-SL-C16	—
2	PUBLIC-FACING CLAIM	Responding to “The Leaderboard Illusion”, LMArena said providers may test several private variants, promised to mark scores provisional until 2,000 fresh votes when more than 10 variants are tested, and disputed the paper’s figures, estimating the gain from private testing at about 11 Elo points; the paper’s 8.8% open-source figure that it disputed was a share of data, not of models.	—
3	CLAIM TYPE	FACT + INTERPRETATION	base_types: FACT, INTERPRETATION
4	RESEARCH STRAND	B — Leaderboards, judges and disclosure	—
5	MATURITY	CONTESTED	states: CONTESTED
6	SOURCE	LMArena, “Our Response to ‘The Leaderboard Illusion’ Writeup,” Arena blog, 9 May 2025 (updated 13 June 2025)	source_ids: SRC-SL-LMARENA-RESPONSE-LEADERBOARD-ILLUSION-2025-05-09
7	SOURCE TYPE	Platform operator blog post	classes: VENDOR_PRIMARY
8	INSTITUTIONAL POSITION	Platform operator (LMArena), the subject of the criticism	—
9	SCOPE	Chatbot Arena’s policies and the disputed figures	—
10	DESIGN	Rebuttal post; compared by SocioLens with the paper’s v1 and v2	—
11	SAMPLE / DATASET	Not applicable	—
12	MODEL / TOOL / VERSION	Not applicable	—
13	COMPARATOR	The paper’s claims (LD06-SL-C15)	—
14	MAIN RESULT	Private variants allowed and to be stated explicitly; clearer marking of retired models; scores provisional until 2,000 fresh votes when more than 10 variants are tested; its own estimate of about +11 Elo after 50 tests and 3,000 votes; the paper’s 112% gain was on Arena-Hard, which the authors’ v2 also says; open models 40.9% on the leaderboard against the paper’s data share	—
15	INFERENTIAL TARGET	How the platform answers the critique and changes its rules	—
16	DOES NOT LICENSE	A finding on which side’s estimate of the private-testing effect is right	—
17	VERIFICATION STATE	CONTENT VERIFIED	state: CONTENT_VERIFIED_PRIMARY · verified_by: UNKNOWN
18	LIMITATIONS	An interested party’s account; the comparison of the two texts is SocioLens’s reading; neither estimate was re-derived	—
19	CONTRADICTORY / SUPPORTING SOURCES	LD06-SL-C15	—
20	GEOGRAPHY / LANGUAGE	Global platform / English	—
21	DISTRIBUTIONAL LEDGER	Not analysed in this claim	—
22	R01 / FIELD-MAP LINKS	LD-06, D4b	—
23	WHAT WOULD CHANGE THIS CLAIM?	An independent re-analysis of Arena data	—
24	PUBLICATION STATUS	ELIGIBLE WITH SOURCE-POSITION QUALIFIER	state: ELIGIBLE · qualifiers: SOURCE_POSITION
25	RECORDED DATE	2026-10-15	—

Field 22 references resolved against Field Map v0.2: LD-06 (LIVING_DEBATE); D4b (SUB_AREA)

The sources as recorded, with their record IDs and coded source types:

Pinned Sources (field 6, version-specific)

“LMArena, “Our Response to ‘The Leaderboard Illusion’ Writeup,” Arena blog, 9 May 2025 (updated 13 June 2025)” → LMArena (2025), “Our Response to ‘The Leaderboard…” (SRC-SL-LMARENA-RESPONSE-LEADERBOARD-ILLUSION-2025-05-09--v1.0) — LMArena, “Our Response to ‘The Leaderboard Illusion’ Writeup,” Arena blog, 9 May 2025 (updated 13 June 2025); class VENDOR_PRIMARY (explicit public provenance pin)

Cited by Living Debate versions

The evaluation apparatus: Can benchmarks, leaderboards and audits carry the weight placed on them? (v0.1) LD-06

Cite this page

SocioLens project. (2026, October 15). Responding to “The Leaderboard Illusion”, LMArena said providers may test several private variants, promised to mark scores provisional until 2,000 fresh votes when more than 10 variants are tested, and disputed the paper’s figures, estimating the gain from private testing at about 11 Elo points; the paper’s 8.8% open-source figure that it disputed was a share of data, not of models [Claim LD06-SL-C16, version v1.0]. SocioLens. https://sociolens-web.pages.dev/claims/ld06-sl-c16/v1.0/

@misc{sociolens_ld06_sl_c16_v1_0,
  author = {{SocioLens project}},
  title = {Responding to “The Leaderboard Illusion”, LMArena said providers may test several private variants, promised to mark scores provisional until 2,000 fresh votes when more than 10 variants are tested, and disputed the paper’s figures, estimating the gain from private testing at about 11 Elo points; the paper’s 8.8\% open-source figure that it disputed was a share of data, not of models.},
  year = {2026},
  date = {2026-10-15},
  howpublished = {\url{https://sociolens-web.pages.dev/claims/ld06-sl-c16/v1.0/}},
  note = {Claim LD06-SL-C16, version v1.0}
}

The link points to this version, whose record never changes. For a finding, cite the original study as well: SocioLens summarizes research and does not replace it.

Research-of-record provenance

Materialized by sociolens-materializer 0.6.0 at 2026-09-28T20:00:00.000Z; normalization contract SocioLens — Claim Value Vocabulary v1.0.2 — Pre-Spike Materialization Contract (v1.0.2). Content seal SHA-256 3fa8acabb4236748fd2ba4dd41ae5a7002b29b94ddcf50a1681b3b22b289d90a.

Repository-authored inputs (not Drive exports)

SocioLens claim — LD06-SL-C16 v1.0 (v1.0) — written in the SocioLens repository by Project Lead (Claude, AI) on 2026-10-15; no Drive identity exists. Authority: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts. Block “ENTRY 16 — LD06-SL-C16” SHA-256 f06a2973673f48ebd79ab4e60cd3b0f050a10e60a345c05adc0117283bf49b11; fixture fixtures/authored/claims/LD06-SL-C16--v1.0.

Builder decisions (not Drive data)

VERSION = v1.0 — basis: Claim version taken from the repository-authored claim fixture version (SocioLens claim — LD06-SL-C16 v1.0); the first version of a new Claim of the LD-06 debate
CURRENT_POINTER = true — basis: repository-authored claim "SocioLens claim — LD06-SL-C16 v1.0" (Project Lead (Claude, AI), 2026-10-15); basis: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts
FIELD_6_SOURCE_REFERENCE = SRC-SL-LMARENA-RESPONSE-LEADERBOARD-ILLUSION-2025-05-09--v1.0 — basis: field 6 of ENTRY 16 — LD06-SL-C16 cites "LMArena, “Our Response to ‘The Leaderboard Illusion’ Writeup,” Arena blog, 9 May 2025 (updated 13 June 2025)"; the only field-6 segment of this ClaimVersion's repository-authored ledger block.
SOURCE_PROVENANCE_PIN = SRC-SL-LMARENA-RESPONSE-LEADERBOARD-ILLUSION-2025-05-09--v1.0 — basis: explicit version-pinned public provenance of this ClaimVersion (Amendment v1.2 §6): the pinned field-6 Source record "LMArena, “Our Response to ‘The Leaderboard Illusion’ Writeup,” Arena blog, 9 May 2025 (updated 13 June 2025)".