LLM simulations can usefully forecast the relative pattern of treatment effects in some US survey-experiment archives, while systematically overstating effect magnitudes.

SocioLens project,

Claim LD07-HQ-C05 (v1.0)

This is an older version. Read the current version, v1.1, or what changed in it.

LLM simulations can usefully forecast the relative pattern of treatment effects in some US survey-experiment archives, while systematically overstating effect magnitudes.

Provenance and status

Claim type: Fact
Publication status: Eligible for publication, with qualifiers: estimand
Verification state: Checked against the primary source; cleared with a source-specific qualifier (recorded as “CLEARED WITH QUALIFIER”). Who checked this claim is not recorded for it; claim checks so far were AI-led, and no person has checked claims against their sources.
Does not license: Synthetic population substitution; unbiased effect sizes; mechanism recovery; subgroup validity; non-US transport
Limitations: US survey-experiment archive; exact public r=.85 attribution to VOR remains HOLD
What this line marks HOLD was revisited in a newer version of this claim, which gives the figure and says what it was checked against: version v1.1.
Source (as cited): Ashokkumar, Hewitt, Ghezae & Willer, Nature 656:115–122 (2026), doi:10.1038/s41586-026-10742-x
Source type: Peer-reviewed journal article (recorded as “Peer-reviewed journal article”)
Institutional position: Independent academic; one author also industry-affiliated
Maturity of the evidence: Contested evidence (positive bounded use)
Recorded date: 25 September 2026
Currentness / correction: This is not the current version. No correction or concern is recorded on its sources.

Sources this version cites

Ashokkumar, Hewitt, Ghezae & Willer, “Large language models can predict the results of social science experiments,” Nature 656:115–122 (2026) (Peer-reviewed journal article)
Read the original

Version

This page shows version v1.0. Its record never changes, even when newer evidence arrives; the site may add a note to the page or withhold a figure not cleared for public use.

NEWER VERSION EXISTS: read the current version, v1.1 (LD07-HQ-C05--v1.1). This version is explicitly superseded by LD07-HQ-C05--v1.1.

Versions: v1.0, v1.1 · current

Raw record: every recorded field, with its coded values

All 25 canonical Claim fields, verbatim, with the coded values derived from them. The plain-language labels above are read from these.

Claim type (coded): FACT (as printed: “FACT”)
Publication status (coded): ELIGIBLE (ESTIMAND) (as printed: “ELIGIBLE WITH ESTIMAND QUALIFIER”)
Verification state (coded): CONTENT_VERIFIED_PRIMARY (SOURCE_SPECIFIC)
Source type (coded): PEER_REVIEWED_JOURNAL_ARTICLE
Maturity (coded): CONTESTED (POSITIVE_BOUNDED_USE)

#	Canonical field	Canonical value (verbatim)	Normalized (Claim Value Vocabulary v1.0.2)
1	CLAIM ID	LD07-HQ-C05	—
2	PUBLIC-FACING CLAIM	LLM simulations can usefully forecast the relative pattern of treatment effects in some US survey-experiment archives, while systematically overstating effect magnitudes.	—
3	CLAIM TYPE	FACT	base_types: FACT
4	RESEARCH STRAND	C — Synthetic respondents/agents	—
5	MATURITY	CONTESTED / POSITIVE BOUNDED USE	states: CONTESTED · qualifiers: POSITIVE_BOUNDED_USE
6	SOURCE	Ashokkumar, Hewitt, Ghezae & Willer, Nature 656:115–122 (2026), doi:10.1038/s41586-026-10742-x	source_ids: SRC-LD07-ASHOKKUMAR-NATURE-2026
7	SOURCE TYPE	Peer-reviewed journal article	classes: PEER_REVIEWED_JOURNAL_ARTICLE
8	INSTITUTIONAL POSITION	Independent academic; one author also industry-affiliated	—
9	SCOPE	US nationally representative text-based survey experiments	—
10	DESIGN	Forecast/model simulation benchmark against realized experimental effects and human forecasts	—
11	SAMPLE / DATASET	70 experiments; 469 effects; 119,330 participants; 15 secondary megastudies/606 effects	—
12	MODEL / TOOL / VERSION	GPT-4 primary plus prominent open-weight models	—
13	COMPARATOR	Realized human treatment effects; pooled human/expert forecasts	—
14	MAIN RESULT	Strong effect-pattern correlation / human-forecast-comparable accuracy; systematic magnitude overestimation	—
15	INFERENTIAL TARGET	Aggregate effect forecasting / replication triage	—
16	DOES NOT LICENSE	Synthetic population substitution; unbiased effect sizes; mechanism recovery; subgroup validity; non-US transport	—
17	VERIFICATION STATE	CLEARED WITH QUALIFIER	state: CONTENT_VERIFIED_PRIMARY · qualifiers: SOURCE_SPECIFIC · verified_by: UNKNOWN
18	LIMITATIONS	US survey-experiment archive; exact public r=.85 attribution to VOR remains HOLD	—
19	CONTRADICTORY / SUPPORTING SOURCES	Li & Ji shows realism ≠ ATE estimation; Cui/Li/Zhou shows null→significant inflation	—
20	GEOGRAPHY / LANGUAGE	US / English	—
21	DISTRIBUTIONAL LEDGER	Subgroup performance unresolved	—
22	R01 / FIELD-MAP LINKS	D9, R-1, T1	—
23	WHAT WOULD CHANGE THIS CLAIM?	Failure on independent non-US experimental archives	—
24	PUBLICATION STATUS	ELIGIBLE WITH ESTIMAND QUALIFIER	state: ELIGIBLE · qualifiers: ESTIMAND
25	RECORDED DATE	2026-09-25	—

Field 22 references resolved against Field Map v0.2: D9 (DOMAIN); R-1 (LENS); T1 (TENSION)

The sources as recorded, with their record IDs and coded source types:

Pinned Sources (field 6, version-specific)

“Ashokkumar, Hewitt, Ghezae & Willer, Nature 656:115–122 (2026), doi:10.1038/s41586-026-10742-x” → Ashokkumar et al. (2026), “Large language models can…” (SRC-LD07-ASHOKKUMAR-NATURE-2026--v1.0) — Ashokkumar, Hewitt, Ghezae & Willer, “Large language models can predict the results of social science experiments,” Nature 656:115–122 (2026); class PEER_REVIEWED_JOURNAL_ARTICLE (explicit public provenance pin)

Cited by Living Debate versions

AI in the research apparatus (v0.1) LD-07
AI in the research apparatus (v0.2) LD-07

Cite this page

SocioLens project. (2026, September 25). LLM simulations can usefully forecast the relative pattern of treatment effects in some US survey-experiment archives, while systematically overstating effect magnitudes [Claim LD07-HQ-C05, version v1.0]. SocioLens. https://sociolens-web.pages.dev/claims/ld07-hq-c05/v1.0/

@misc{sociolens_ld07_hq_c05_v1_0,
  author = {{SocioLens project}},
  title = {LLM simulations can usefully forecast the relative pattern of treatment effects in some US survey-experiment archives, while systematically overstating effect magnitudes.},
  year = {2026},
  date = {2026-09-25},
  howpublished = {\url{https://sociolens-web.pages.dev/claims/ld07-hq-c05/v1.0/}},
  note = {Claim LD07-HQ-C05, version v1.0}
}

The link points to this version, whose record never changes. For a finding, cite the original study as well: SocioLens summarizes research and does not replace it.

Research-of-record provenance

Materialized by sociolens-materializer 0.6.0 at 2026-09-28T20:00:00.000Z; normalization contract SocioLens — Claim Value Vocabulary v1.0.2 — Pre-Spike Materialization Contract (v1.0.2). Content seal SHA-256 3f71c4aeda5109c5ef8070a2dfe6de31c4e43804cd447644e0b4164f618228e5.

Frozen Drive-derived inputs

R03 — LD-07 Canonical Claim Ledger v1.0 — LOCKED (v1.0) — Drive ID 1xVz_LrQAA8EA1tPhVWIrDgOUU3Zidlg9; revision: NOT_EXPOSED_BY_DRIVE_TEXT_EXPORT; Drive modified 2026-09-29T04:31:14.120Z; block “ENTRY 05 — LD07-HQ-C05” SHA-256 1bde3e034b0890c118453eca52b1332ac22679f04989cf11b3a217a52a4a8f8c; SocioLens — Programme Status & Authority Register v0.6 status “CURRENT.”; frozen fixture fixtures/frozen/r03-ld07-claim-ledger-v1.0-canonical.

Builder decisions (not Drive data)

VERSION = v1.0 — basis: Claim version taken from the canonical ledger artifact version (R03 — LD-07 Canonical Claim Ledger v1.0); Implementation Architecture Selection §5
CURRENT_POINTER = false — basis: not current: superseded by LD07-HQ-C05--v1.1 (manifest supersedes relation)
FIELD_17_CONTEXT_QUALIFIERS = SOURCE_SPECIFIC — basis: R03 — HQ Cross-Batch Synthesis & Adjudication (v0.1, Drive 1V-HOVaPFwF8BTf60-LGPsJXf59V3kRlQ)
FIELD_6_SOURCE_REFERENCE = SRC-LD07-ASHOKKUMAR-NATURE-2026--v1.0 — basis: field 6 of ENTRY 05 — LD07-HQ-C05 cites "Ashokkumar, Hewitt, Ghezae & Willer, Nature 656:115–122 (2026), doi:10.1038/s41586-026-10742-x"; the frozen field-6 segment of this ClaimVersion's ledger block; Drive-derived identity authority (Issue #39 Source inventory): Batch 02 §1; ledger C05 fields 6/7/8; KEY EVIDENCE.
SOURCE_PROVENANCE_PIN = SRC-LD07-ASHOKKUMAR-NATURE-2026--v1.0 — basis: Issue #5 proof rendering (Architecture Proof Protocol v0.1 AP-08 / Amendment v1.2 §6): explicit version-pinned public provenance of this ClaimVersion — the pinned field-6 Source record "Ashokkumar, Hewitt, Ghezae & Willer, Nature 656:115–122 (2026), doi:10.1038/s41586-026-10742-x"; Drive-derived identity authority (Issue #39 Source inventory): Batch 02 §1; ledger C05 fields 6/7/8; KEY EVIDENCE.