Across more than 350 language models, when two models both answer a leaderboard question wrongly they give the same wrong answer 60% of the time on one dataset, and larger, more accurate models have highly correlated errors even across different architectures and providers.

SocioLens project,

Claim LD09-SL-C25 (v1.0)

Research opportunities this claim motivates

What this claim leaves open for research. Each is a hypothesis about a gap, not a finding.

Does pooling different models keep restoring diversity as models converge?Larger, more accurate models have highly correlated errors across architectures and providers.
Does concentration at the model layer make apps fail alike? A test on China's registerAcross more than 350 models, errors are correlated on leaderboard questions, highly so among larger models even across providers; this is the baseline a shared-model effect must exceed, and the claim does not reach effects on real decisions.

Provenance and status

Claim type: Estimate
Publication status: Eligible for publication, with a qualifier
Verification state: Checked against the abstract of the primary source only. Who checked this claim is not recorded for it; claim checks so far were AI-led, and no person has checked claims against their sources.
Does not license: Effects on real decisions
Limitations: Official abstract only; benchmark tasks; agreement baselines matter (LD09-SL-C26)
Source (as cited): Kim, Garg, Peng & Garg, “Correlated Errors in Large Language Models,” ICML 2025, arXiv:2506.07962
Source type: Peer-reviewed conference paper (recorded as “Peer-reviewed conference paper”)
Institutional position: Independent academic
Maturity of the evidence: Emerging evidence
Recorded date: 15 October 2026
Currentness / correction: This is the current version. No correction or concern is recorded on its sources.

Sources this version cites

Kim, Garg, Peng & Garg, “Correlated Errors in Large Language Models,” ICML 2025, arXiv:2506.07962 (Peer-reviewed conference paper)
Read the original

Version

This is the current version, v1.0. Earlier versions stay available, and their records never change.

Versions: v1.0 · current

Raw record: every recorded field, with its coded values

All 25 canonical Claim fields, verbatim, with the coded values derived from them. The plain-language labels above are read from these.

Claim type (coded): ESTIMATE (as printed: “ESTIMATE”)
Publication status (coded): ELIGIBLE (QUALIFIED) (as printed: “ELIGIBLE WITH QUALIFIER”)
Verification state (coded): CONTENT_VERIFIED_PRIMARY (ABSTRACT_ONLY)
Source type (coded): PEER_REVIEWED_CONFERENCE_PAPER
Maturity (coded): EMERGING

#	Canonical field	Canonical value (verbatim)	Normalized (Claim Value Vocabulary v1.0.2)
1	CLAIM ID	LD09-SL-C25	—
2	PUBLIC-FACING CLAIM	Across more than 350 language models, when two models both answer a leaderboard question wrongly they give the same wrong answer 60% of the time on one dataset, and larger, more accurate models have highly correlated errors even across different architectures and providers.	—
3	CLAIM TYPE	ESTIMATE	base_types: ESTIMATE
4	RESEARCH STRAND	C — Systems, decisions and shared knowledge	—
5	MATURITY	EMERGING	states: EMERGING
6	SOURCE	Kim, Garg, Peng & Garg, “Correlated Errors in Large Language Models,” ICML 2025, arXiv:2506.07962	source_ids: SRC-SL-KIM-ET-AL-CORRELATED-ERRORS-ICML-2025
7	SOURCE TYPE	Peer-reviewed conference paper	classes: PEER_REVIEWED_CONFERENCE_PAPER
8	INSTITUTIONAL POSITION	Independent academic	—
9	SCOPE	Leaderboard tasks and a résumé-screening task	—
10	DESIGN	Analysis of agreement between models’ errors	—
11	SAMPLE / DATASET	More than 350 language models on two leaderboards	—
12	MODEL / TOOL / VERSION	Language models from many providers	—
13	COMPARATOR	Pairs of models from different architectures and providers	—
14	MAIN RESULT	60% agreement when both err on one dataset; correlation higher among larger, more accurate models; downstream effects in model-as-judge and hiring settings	—
15	INFERENTIAL TARGET	Whether using models from different providers avoids shared errors	—
16	DOES NOT LICENSE	Effects on real decisions	—
17	VERIFICATION STATE	CONTENT VERIFIED ABSTRACT	state: CONTENT_VERIFIED_PRIMARY · qualifiers: ABSTRACT_ONLY · verified_by: UNKNOWN
18	LIMITATIONS	Official abstract only; benchmark tasks; agreement baselines matter (LD09-SL-C26)	—
19	CONTRADICTORY / SUPPORTING SOURCES	Diversity restored by pooling vendors in idea generation: LD09-SL-C17	—
20	GEOGRAPHY / LANGUAGE	Not checked / English	—
21	DISTRIBUTIONAL LEDGER	Not recorded	—
22	R01 / FIELD-MAP LINKS	LD-09, D3	—
23	WHAT WOULD CHANGE THIS CLAIM?	Evidence that model diversity reduces shared errors in deployed decisions	—
24	PUBLICATION STATUS	ELIGIBLE WITH QUALIFIER	state: ELIGIBLE · qualifiers: QUALIFIED
25	RECORDED DATE	2026-10-15	—

Field 22 references resolved against Field Map v0.2: LD-09 (LIVING_DEBATE); D3 (DOMAIN)

The sources as recorded, with their record IDs and coded source types:

Pinned Sources (field 6, version-specific)

“Kim, Garg, Peng & Garg, “Correlated Errors in Large Language Models,” ICML 2025, arXiv:2506.07962” → Kim et al. (2025), “Correlated Errors in Large Language…” (SRC-SL-KIM-ET-AL-CORRELATED-ERRORS-ICML-2025--v1.0) — Kim, Garg, Peng & Garg, “Correlated Errors in Large Language Models,” ICML 2025, arXiv:2506.07962; class PEER_REVIEWED_CONFERENCE_PAPER (explicit public provenance pin)

Cited by Living Debate versions

Individual gain, collective narrowing? Does AI make science, creative work and shared knowledge more alike? (v0.1) LD-09

Cite this page

SocioLens project. (2026, October 15). Across more than 350 language models, when two models both answer a leaderboard question wrongly they give the same wrong answer 60% of the time on one dataset, and larger, more accurate models have highly correlated errors even across different architectures and providers [Claim LD09-SL-C25, version v1.0]. SocioLens. https://sociolens-web.pages.dev/claims/ld09-sl-c25/v1.0/

@misc{sociolens_ld09_sl_c25_v1_0,
  author = {{SocioLens project}},
  title = {Across more than 350 language models, when two models both answer a leaderboard question wrongly they give the same wrong answer 60\% of the time on one dataset, and larger, more accurate models have highly correlated errors even across different architectures and providers.},
  year = {2026},
  date = {2026-10-15},
  howpublished = {\url{https://sociolens-web.pages.dev/claims/ld09-sl-c25/v1.0/}},
  note = {Claim LD09-SL-C25, version v1.0}
}

The link points to this version, whose record never changes. For a finding, cite the original study as well: SocioLens summarizes research and does not replace it.

Research-of-record provenance

Materialized by sociolens-materializer 0.6.0 at 2026-09-28T20:00:00.000Z; normalization contract SocioLens — Claim Value Vocabulary v1.0.2 — Pre-Spike Materialization Contract (v1.0.2). Content seal SHA-256 d2ba0afec675d4cecc1994c601db448c669bd305c96057e5a73258d141d983cc.

Repository-authored inputs (not Drive exports)

SocioLens claim — LD09-SL-C25 v1.0 (v1.0) — written in the SocioLens repository by Project Lead (Claude, AI) on 2026-10-15; no Drive identity exists. Authority: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-09 debate; a new Claim, first version, written from the project's LD-09 source notes, checked against full texts or official abstracts. Block “ENTRY 25 — LD09-SL-C25” SHA-256 7e1921d65516278e0128ddb0d71cc5b3a06746fe97bc2a1742a6ea27aa0e4555; fixture fixtures/authored/claims/LD09-SL-C25--v1.0.

Builder decisions (not Drive data)

VERSION = v1.0 — basis: Claim version taken from the repository-authored claim fixture version (SocioLens claim — LD09-SL-C25 v1.0); the first version of a new Claim of the LD-09 debate
CURRENT_POINTER = true — basis: repository-authored claim "SocioLens claim — LD09-SL-C25 v1.0" (Project Lead (Claude, AI), 2026-10-15); basis: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-09 debate; a new Claim, first version, written from the project's LD-09 source notes, checked against full texts or official abstracts
FIELD_6_SOURCE_REFERENCE = SRC-SL-KIM-ET-AL-CORRELATED-ERRORS-ICML-2025--v1.0 — basis: field 6 of ENTRY 25 — LD09-SL-C25 cites "Kim, Garg, Peng & Garg, “Correlated Errors in Large Language Models,” ICML 2025, arXiv:2506.07962"; the only field-6 segment of this ClaimVersion's repository-authored ledger block.
SOURCE_PROVENANCE_PIN = SRC-SL-KIM-ET-AL-CORRELATED-ERRORS-ICML-2025--v1.0 — basis: explicit version-pinned public provenance of this ClaimVersion (Amendment v1.2 §6): the pinned field-6 Source record "Kim, Garg, Peng & Garg, “Correlated Errors in Large Language Models,” ICML 2025, arXiv:2506.07962".