In a large 2025 preprint spanning 37 annotation tasks from 21 published studies, plausible choices of LLM and annotation configuration produced incorrect inferential conclusions in about one third of tested hypotheses for state-of-the-art models, with higher rates for smaller models and near conventional significance thresholds.

SocioLens project,

Claim LD07-HQ-C02 (v1.0)

Research opportunities this claim motivates

What this claim leaves open for research. Each is a hypothesis about a gap, not a finding.

Do pinned model snapshots keep social-science conclusions stable over time?Configuration choices already flip about a third of tested conclusions at a single point in time.
Does quantization or the serving engine change social-science findings?Analytic flexibility in model configuration already changes inferential conclusions; deployment settings are an unmeasured extension of that configuration space.

Provenance and status

Claim type: Estimate
Publication status: Eligible for publication, with qualifiers: preprint, corpus
Verification state: Checked against the primary source; the source is a preprint; cleared with a source-specific qualifier (recorded as “CLEARED WITH QUALIFIER”). Who checked this claim is not recorded for it; claim checks so far were AI-led, and no person has checked claims against their sources.
Does not license: “31% of published findings are wrong”; future-model extrapolation
Limitations: Preprint; corpus/model-generation specific; configuration space defined by authors
Source (as cited): Baumann, Röttger, Urman, Wendsjö, Plaza-del-Arco, Gruber & Hovy, “Large Language Model Hacking,” arXiv:2509.08825v2
Source type: Preprint (recorded as “Preprint”)
Institutional position: Independent academic
Maturity of the evidence: Rapidly maturing evidence
Recorded date: 25 September 2026
Currentness / correction: This is the current version. No correction or concern is recorded on its sources.

Sources this version cites

Baumann, Röttger, Urman, Wendsjö, Plaza-del-Arco, Gruber & Hovy, “Large Language Model Hacking,” arXiv:2509.08825v2 (Preprint)
Read the original

Version

This is the current version, v1.0. Earlier versions stay available, and their records never change.

Versions: v1.0 · current

Raw record: every recorded field, with its coded values

All 25 canonical Claim fields, verbatim, with the coded values derived from them. The plain-language labels above are read from these.

Claim type (coded): ESTIMATE (as printed: “ESTIMATE”)
Publication status (coded): ELIGIBLE (PREPRINT, CORPUS) (as printed: “ELIGIBLE WITH PREPRINT/CORPUS QUALIFIER”)
Verification state (coded): CONTENT_VERIFIED_PRIMARY (PREPRINT, SOURCE_SPECIFIC)
Source type (coded): PREPRINT
Maturity (coded): RAPIDLY_MATURING

#	Canonical field	Canonical value (verbatim)	Normalized (Claim Value Vocabulary v1.0.2)
1	CLAIM ID	LD07-HQ-C02	—
2	PUBLIC-FACING CLAIM	In a large 2025 preprint spanning 37 annotation tasks from 21 published studies, plausible choices of LLM and annotation configuration produced incorrect inferential conclusions in about one third of tested hypotheses for state-of-the-art models, with higher rates for smaller models and near conventional significance thresholds.	—
3	CLAIM TYPE	ESTIMATE	base_types: ESTIMATE
4	RESEARCH STRAND	A	—
5	MATURITY	RAPIDLY MATURING	states: RAPIDLY_MATURING
6	SOURCE	Baumann, Röttger, Urman, Wendsjö, Plaza-del-Arco, Gruber & Hovy, “Large Language Model Hacking,” arXiv:2509.08825v2	source_ids: SRC-LD07-BAUMANN-LLM-HACKING-2025
7	SOURCE TYPE	Preprint	classes: PREPRINT
8	INSTITUTIONAL POSITION	Independent academic	—
9	SCOPE	37 CSS annotation tasks from 21 studies	—
10	DESIGN	Large multi-study re-annotation/configuration audit	—
11	SAMPLE / DATASET	~13M labels; 2,361 realistic hypotheses	—
12	MODEL / TOOL / VERSION	18 LLMs, 2025-era model set	—
13	COMPARATOR	Study-specific ground-truth/reference annotation and hypothesis conclusions	—
14	MAIN RESULT	~31% incorrect conclusions for SOTA; ~50% smaller models; risk higher near p=.05	—
15	INFERENTIAL TARGET	Preservation of statistical conclusions under annotation configuration choice	—
16	DOES NOT LICENSE	“31% of published findings are wrong”; future-model extrapolation	—
17	VERIFICATION STATE	CLEARED WITH QUALIFIER	state: CONTENT_VERIFIED_PRIMARY · qualifiers: PREPRINT, SOURCE_SPECIFIC · verified_by: UNKNOWN
18	LIMITATIONS	Preprint; corpus/model-generation specific; configuration space defined by authors	—
19	CONTRADICTORY / SUPPORTING SOURCES	Egami; Yang; Li; Nakamura qualifies task-level agreement interpretation	—
20	GEOGRAPHY / LANGUAGE	Mostly English CSS	—
21	DISTRIBUTIONAL LEDGER	No broad subgroup guarantee	—
22	R01 / FIELD-MAP LINKS	T1, D9, R-1, R-2	—
23	WHAT WOULD CHANGE THIS CLAIM?	Peer-reviewed replication with current models and substantially lower error	—
24	PUBLICATION STATUS	ELIGIBLE WITH PREPRINT/CORPUS QUALIFIER	state: ELIGIBLE · qualifiers: PREPRINT, CORPUS
25	RECORDED DATE	2026-09-25	—

Field 22 references resolved against Field Map v0.2: T1 (TENSION); D9 (DOMAIN); R-1 (LENS); R-2 (LENS)

The sources as recorded, with their record IDs and coded source types:

Pinned Sources (field 6, version-specific)

“Baumann, Röttger, Urman, Wendsjö, Plaza-del-Arco, Gruber & Hovy, “Large Language Model Hacking,” arXiv:2509.08825v2” → Baumann et al., “Large Language Model Hacking” (SRC-LD07-BAUMANN-LLM-HACKING-2025--v1.0) — Baumann, Röttger, Urman, Wendsjö, Plaza-del-Arco, Gruber & Hovy, “Large Language Model Hacking,” arXiv:2509.08825v2; class PREPRINT (explicit public provenance pin)

Cited by Living Debate versions

AI in the research apparatus (v0.1) LD-07
AI in the research apparatus (v0.2) LD-07

Cite this page

SocioLens project. (2026, September 25). In a large 2025 preprint spanning 37 annotation tasks from 21 published studies, plausible choices of LLM and annotation configuration produced incorrect inferential conclusions in about one third of tested hypotheses for state-of-the-art models, with higher rates for smaller models and near conventional significance thresholds [Claim LD07-HQ-C02, version v1.0]. SocioLens. https://sociolens-web.pages.dev/claims/ld07-hq-c02/v1.0/

@misc{sociolens_ld07_hq_c02_v1_0,
  author = {{SocioLens project}},
  title = {In a large 2025 preprint spanning 37 annotation tasks from 21 published studies, plausible choices of LLM and annotation configuration produced incorrect inferential conclusions in about one third of tested hypotheses for state-of-the-art models, with higher rates for smaller models and near conventional significance thresholds.},
  year = {2026},
  date = {2026-09-25},
  howpublished = {\url{https://sociolens-web.pages.dev/claims/ld07-hq-c02/v1.0/}},
  note = {Claim LD07-HQ-C02, version v1.0}
}

The link points to this version, whose record never changes. For a finding, cite the original study as well: SocioLens summarizes research and does not replace it.

Research-of-record provenance

Materialized by sociolens-materializer 0.6.0 at 2026-09-28T20:00:00.000Z; normalization contract SocioLens — Claim Value Vocabulary v1.0.2 — Pre-Spike Materialization Contract (v1.0.2). Content seal SHA-256 305c71041ba2d8d1231b52858ca7825ff07748b3081aa34b3c739dcfce4d0145.

Frozen Drive-derived inputs

R03 — LD-07 Canonical Claim Ledger v1.0 — LOCKED (v1.0) — Drive ID 1xVz_LrQAA8EA1tPhVWIrDgOUU3Zidlg9; revision: NOT_EXPOSED_BY_DRIVE_TEXT_EXPORT; Drive modified 2026-09-29T04:31:14.120Z; block “ENTRY 02 — LD07-HQ-C02” SHA-256 d7b3ad85094278da9806fd9b7ff86c521940f9960c462480c6b32801419f55b1; SocioLens — Programme Status & Authority Register v0.6 status “CURRENT.”; frozen fixture fixtures/frozen/r03-ld07-claim-ledger-v1.0-canonical.

Builder decisions (not Drive data)

VERSION = v1.0 — basis: Claim version taken from the canonical ledger artifact version (R03 — LD-07 Canonical Claim Ledger v1.0); Implementation Architecture Selection §5
CURRENT_POINTER = true — basis: SocioLens — Programme Status & Authority Register v0.6 status for "R03 — LD-07 Canonical Claim Ledger v1.0 — LOCKED": "CURRENT."
FIELD_17_CONTEXT_QUALIFIERS = PREPRINT, SOURCE_SPECIFIC — basis: R03 — HQ Cross-Batch Synthesis & Adjudication (v0.1, Drive 1V-HOVaPFwF8BTf60-LGPsJXf59V3kRlQ)
FIELD_6_SOURCE_REFERENCE = SRC-LD07-BAUMANN-LLM-HACKING-2025--v1.0 — basis: field 6 of ENTRY 02 — LD07-HQ-C02 cites "Baumann, Röttger, Urman, Wendsjö, Plaza-del-Arco, Gruber & Hovy, “Large Language Model Hacking,” arXiv:2509.08825v2"; the frozen field-6 segment of this ClaimVersion's ledger block; Drive-derived identity authority (Issue #39 Source inventory): Batch 01 §2 ("PREPRINT", v2); ledger C02; KEY EVIDENCE.
SOURCE_PROVENANCE_PIN = SRC-LD07-BAUMANN-LLM-HACKING-2025--v1.0 — basis: Issue #5 proof rendering (Architecture Proof Protocol v0.1 AP-08 / Amendment v1.2 §6): explicit version-pinned public provenance of this ClaimVersion — the pinned field-6 Source record "Baumann, Röttger, Urman, Wendsjö, Plaza-del-Arco, Gruber & Hovy, “Large Language Model Hacking,” arXiv:2509.08825v2"; Drive-derived identity authority (Issue #39 Source inventory): Batch 01 §2 ("PREPRINT", v2); ledger C02; KEY EVIDENCE.