Claim LD07-HQ-C06 (v1.0)
Synthetic respondents can match selected survey averages while failing to reproduce real-world variance, regression relationships, individual structure and subgroup heterogeneity.
Research opportunities this claim motivates
What this claim leaves open for research. Each is a hypothesis about a gap, not a finding.
- Can synthetic respondents recover strong treatment-effect heterogeneity?Synthetic samples can match selected averages while failing to reproduce variance and subgroup heterogeneity.
- How do research ethics boards judge synthetic participants and AI-interpreted data?Synthetic respondents do not substitute for populations, which bears on whether they reduce or relocate research risk.
- Do silicon samples work outside the North Atlantic, and in which language?Subgroup and variance failures are documented in the contexts tested so far.
- Does asking in Greek change how AI "silicon samples" answer as Greek respondents?Synthetic respondents can match selected survey averages while failing to reproduce variance, regression relationships and subgroup differences.
Provenance and status
- Claim type
- Fact and Interpretation
- Publication status
- Eligible for publication
- Verification state
- Checked against the primary source. Who checked this claim is not recorded for it; claim checks so far were AI-led, and no person has checked claims against their sources.
- Does not license
- “Synthetic data are never useful”; impossibility of future validated simulators
- Limitations
- Designs and conditioning richness differ across studies
- Source (as cited)
- Bisbee et al. Political Analysis 2024; Xie et al. PNAS 2026; Wang et al. NMI 2025; Gong & Huang 2025; Ling & Cui 2025
- Source type
- Peer-reviewed journal article (recorded as “Multiple peer-reviewed studies”)
- Institutional position
- Independent academic
- Maturity of the evidence
- Contested evidence (negative evidence multi study)
- Recorded date
- 25 September 2026
- Currentness / correction
- This is the current version. No correction or concern is recorded on its sources.
Sources this version cites
- Bisbee, Clinton, Dorff, Kenkel & Larson, “Synthetic Replacements for Human Survey Data? The Perils of Large Language Models,” Political Analysis 32(4):401–416 (2024) (Peer-reviewed journal article)
Read the original - Xie et al., “Evaluating the statistical realism of LLM-generated social science data,” PNAS 123(19):e2538145123 (2026) (Peer-reviewed journal article)
Read the original - Wang et al. NMI 2025 (Peer-reviewed journal article)
No checked link to the original yet. - Gong Weigang & Huang Siyuan, “整体事实还是有偏样本——基于大语言模型生成数据的测量,” Academic Monthly 57(6):123–136 (2025) (Peer-reviewed journal article)
Read the original - Ling Wanying & Cui Sizhan, “复现的限度——‘硅基样本’的可能与可为,” Journal of Intelligent Society 4(2):179–201 (2025) (Peer-reviewed journal article)
No checked link to the original yet.
Version
This is the current version, v1.0. Earlier versions stay available, and their records never change.
Raw record: every recorded field, with its coded values
All 25 canonical Claim fields, verbatim, with the coded values derived from them. The plain-language labels above are read from these.
- Claim type (coded)
- FACT + INTERPRETATION (as printed: “FACT / INTERPRETATION”)
- Publication status (coded)
- ELIGIBLE (as printed: “ELIGIBLE”)
- Verification state (coded)
- CONTENT_VERIFIED_PRIMARY
- Source type (coded)
- PEER_REVIEWED_JOURNAL_ARTICLE
- Maturity (coded)
- CONTESTED (NEGATIVE_EVIDENCE_MULTI_STUDY)
| # | Canonical field | Canonical value (verbatim) | Normalized (Claim Value Vocabulary v1.0.2) |
|---|---|---|---|
| 1 | CLAIM ID | LD07-HQ-C06 | — |
| 2 | PUBLIC-FACING CLAIM | Synthetic respondents can match selected survey averages while failing to reproduce real-world variance, regression relationships, individual structure and subgroup heterogeneity. | — |
| 3 | CLAIM TYPE | FACT / INTERPRETATION | base_types: FACT, INTERPRETATION |
| 4 | RESEARCH STRAND | C | — |
| 5 | MATURITY | CONTESTED, NEGATIVE EVIDENCE MULTI-STUDY | states: CONTESTED · qualifiers: NEGATIVE_EVIDENCE_MULTI_STUDY |
| 6 | SOURCE | Bisbee et al. Political Analysis 2024; Xie et al. PNAS 2026; Wang et al. NMI 2025; Gong & Huang 2025; Ling & Cui 2025 | source_ids: SRC-LD07-BISBEE-SYNTHETIC-REPLACEMENTS-2024, SRC-LD07-XIE-SSDATABENCH-2026, SRC-R03-WANG-MORGENSTERN-DICKERSON-NMI-2025, SRC-LD07-GONG-HUANG-CGSS-2025, SRC-LD07-LING-CUI-SILICON-SAMPLES-2025 |
| 7 | SOURCE TYPE | Multiple peer-reviewed studies | classes: PEER_REVIEWED_JOURNAL_ARTICLE |
| 8 | INSTITUTIONAL POSITION | Independent academic | — |
| 9 | SCOPE | ANES, multi-survey statistical benchmark, identity groups, CGSS | — |
| 10 | DESIGN | Human-vs-synthetic comparisons; statistical realism benchmarks | — |
| 11 | SAMPLE / DATASET | Multiple US/China/UK surveys and identity studies | — |
| 12 | MODEL / TOOL / VERSION | GPT-3.5/4 lineages plus multiple newer/open models | — |
| 13 | COMPARATOR | Human survey/population data | — |
| 14 | MAIN RESULT | Mean/topline match can coexist with compressed variance, altered coefficients, stereotyped/flattened groups and cross-model inconsistency | — |
| 15 | INFERENTIAL TARGET | Population representation and downstream associations | — |
| 16 | DOES NOT LICENSE | “Synthetic data are never useful”; impossibility of future validated simulators | — |
| 17 | VERIFICATION STATE | CLEARED | state: CONTENT_VERIFIED_PRIMARY · verified_by: UNKNOWN |
| 18 | LIMITATIONS | Designs and conditioning richness differ across studies | — |
| 19 | CONTRADICTORY / SUPPORTING SOURCES | Ashokkumar is positive on a different forecasting estimand | — |
| 20 | GEOGRAPHY / LANGUAGE | US, China, UK; English/Chinese | — |
| 21 | DISTRIBUTIONAL LEDGER | Marginal/identity groups particularly vulnerable to flattening/underrepresentation | — |
| 22 | R01 / FIELD-MAP LINKS | L-A, L-B, D9 | — |
| 23 | WHAT WOULD CHANGE THIS CLAIM? | Preregistered subgroup-valid synthetic population benchmark with probability-survey gold | — |
| 24 | PUBLICATION STATUS | ELIGIBLE | state: ELIGIBLE |
| 25 | RECORDED DATE | 2026-09-25 | — |
Field 22 references resolved against Field Map v0.2: L-A (LENS); L-B (LENS); D9 (DOMAIN)
The sources as recorded, with their record IDs and coded source types:
Pinned Sources (field 6, version-specific)
- “Bisbee et al. Political Analysis 2024” → Bisbee et al. (2024), “Synthetic Replacements for Human…” (SRC-LD07-BISBEE-SYNTHETIC-REPLACEMENTS-2024--v1.0) — Bisbee, Clinton, Dorff, Kenkel & Larson, “Synthetic Replacements for Human Survey Data? The Perils of Large Language Models,” Political Analysis 32(4):401–416 (2024); class PEER_REVIEWED_JOURNAL_ARTICLE (explicit public provenance pin)
- “Xie et al. PNAS 2026” → Xie et al. (2026), “Evaluating the statistical realism of…” (SRC-LD07-XIE-SSDATABENCH-2026--v1.0) — Xie et al., “Evaluating the statistical realism of LLM-generated social science data,” PNAS 123(19):e2538145123 (2026); class PEER_REVIEWED_JOURNAL_ARTICLE (explicit public provenance pin)
- “Wang et al. NMI 2025” → Wang et al. NMI 2025 (SRC-R03-WANG-MORGENSTERN-DICKERSON-NMI-2025--v1.0) — Wang et al. NMI 2025; class PEER_REVIEWED_JOURNAL_ARTICLE (explicit public provenance pin)
- “Gong & Huang 2025” → Gong Weigang & Huang Siyuan (2025), “整体事实还是有偏样本——基于大语言模型生成…” (SRC-LD07-GONG-HUANG-CGSS-2025--v1.0) — Gong Weigang & Huang Siyuan, “整体事实还是有偏样本——基于大语言模型生成数据的测量,” Academic Monthly 57(6):123–136 (2025); class PEER_REVIEWED_JOURNAL_ARTICLE (explicit public provenance pin)
- “Ling & Cui 2025” → Ling Wanying & Cui Sizhan (2025), “复现的限度——‘硅基样本’的可能与可为” (SRC-LD07-LING-CUI-SILICON-SAMPLES-2025--v1.0) — Ling Wanying & Cui Sizhan, “复现的限度——‘硅基样本’的可能与可为,” Journal of Intelligent Society 4(2):179–201 (2025); class PEER_REVIEWED_JOURNAL_ARTICLE (explicit public provenance pin)
Cited by Living Debate versions
Radar Updates referencing this ClaimVersion
- Radar update: Pew test of synthetic survey respondents (LD-07) RADAR-LD07-PEW-SYNTHETIC-SURVEYS-2026: Pew Research Center (30 September 2026) built an AI “digital twin” for each panelist analyzed from three 2026 waves of its American Trends Panel, a nationally representative panel of randomly selected U.S. adults. Each twin was conditioned on that person’s profile and earlier survey answers, and weighted synthetic and human results were compared on nearly 300 questions. With its best-performing configuration (Claude Opus 4.6), synthetic estimates differed from human estimates by an average of 12.4 percentage points. Error was substantial for every subgroup reported, reaching 16.1 points for Republicans and Republican leaners and 15.1 points for Black adults. On 47% of questions at least one answer choice was selected by no synthetic respondent, which happened on no question among humans. Results also differed systematically by AI model, although some toplines matched closely. This strengthens the evidence for LD07-HQ-C06 on selected-average agreement, real-world response variance and subgroup heterogeneity. The report contains no regression or treatment-effect analysis and no individual-level accuracy result for its final configuration, so those parts of the claim are unchanged. The claim wording is unchanged.
Cite this page
SocioLens project. (2026, September 25). Synthetic respondents can match selected survey averages while failing to reproduce real-world variance, regression relationships, individual structure and subgroup heterogeneity [Claim LD07-HQ-C06, version v1.0]. SocioLens. https://sociolens-web.pages.dev/claims/ld07-hq-c06/v1.0/
@misc{sociolens_ld07_hq_c06_v1_0,
author = {{SocioLens project}},
title = {Synthetic respondents can match selected survey averages while failing to reproduce real-world variance, regression relationships, individual structure and subgroup heterogeneity.},
year = {2026},
date = {2026-09-25},
howpublished = {\url{https://sociolens-web.pages.dev/claims/ld07-hq-c06/v1.0/}},
note = {Claim LD07-HQ-C06, version v1.0}
}The link points to this version, whose record never changes. For a finding, cite the original study as well: SocioLens summarizes research and does not replace it.
Research-of-record provenance
Materialized by sociolens-materializer 0.6.0 at 2026-09-28T20:00:00.000Z; normalization contract SocioLens — Claim Value Vocabulary v1.0.2 — Pre-Spike Materialization Contract (v1.0.2). Content seal SHA-256 da2589d0c80046281ebe5265eb8fbdd4168d24a03cf90062e8887b9f908894b1.
Frozen Drive-derived inputs
- R03 — LD-07 Canonical Claim Ledger v1.0 — LOCKED (v1.0) — Drive ID
1xVz_LrQAA8EA1tPhVWIrDgOUU3Zidlg9; revision: NOT_EXPOSED_BY_DRIVE_TEXT_EXPORT; Drive modified 2026-09-29T04:31:14.120Z; block “ENTRY 06 — LD07-HQ-C06” SHA-2564bcc6149e6f274dab11470cb90500fc248e0b7858fd5016e2abb8fecf89815e1; SocioLens — Programme Status & Authority Register v0.6 status “CURRENT.”; frozen fixturefixtures/frozen/r03-ld07-claim-ledger-v1.0-canonical.
Builder decisions (not Drive data)
- VERSION = v1.0 — basis: Claim version taken from the canonical ledger artifact version (R03 — LD-07 Canonical Claim Ledger v1.0); Implementation Architecture Selection §5
- CURRENT_POINTER = true — basis: SocioLens — Programme Status & Authority Register v0.6 status for "R03 — LD-07 Canonical Claim Ledger v1.0 — LOCKED": "CURRENT."
- FIELD_6_SOURCE_REFERENCE = SRC-LD07-BISBEE-SYNTHETIC-REPLACEMENTS-2024--v1.0 — basis: field 6 of ENTRY 06 — LD07-HQ-C06 cites "Bisbee et al. Political Analysis 2024"; the frozen field-6 segment of this ClaimVersion's ledger block; Drive-derived identity authority (Issue #39 Source inventory): Batch 02 §2; ledger C06; KEY EVIDENCE.
- FIELD_6_SOURCE_REFERENCE = SRC-LD07-XIE-SSDATABENCH-2026--v1.0 — basis: field 6 of ENTRY 06 — LD07-HQ-C06 cites "Xie et al. PNAS 2026"; the frozen field-6 segment of this ClaimVersion's ledger block; Drive-derived identity authority (Issue #39 Source inventory): Batch 02 §3; ledger C06; KEY EVIDENCE.
- FIELD_6_SOURCE_REFERENCE = SRC-R03-WANG-MORGENSTERN-DICKERSON-NMI-2025--v1.0 — basis: field 6 of ENTRY 06 — LD07-HQ-C06 cites "Wang et al. NMI 2025"; the frozen field-6 segment of this ClaimVersion's ledger block; Central HQ ruling embedded in this issue, resolution ruling §1.
- FIELD_6_SOURCE_REFERENCE = SRC-LD07-GONG-HUANG-CGSS-2025--v1.0 — basis: field 6 of ENTRY 06 — LD07-HQ-C06 cites "Gong & Huang 2025"; the frozen field-6 segment of this ClaimVersion's ledger block; Drive-derived identity authority (Issue #39 Source inventory): Batch 02 §4; ledger C06/C08; KEY EVIDENCE.
- FIELD_6_SOURCE_REFERENCE = SRC-LD07-LING-CUI-SILICON-SAMPLES-2025--v1.0 — basis: field 6 of ENTRY 06 — LD07-HQ-C06 cites "Ling & Cui 2025"; the frozen field-6 segment of this ClaimVersion's ledger block; Drive-derived identity authority (Issue #39 Source inventory): Batch 02 §5 (distinct study); ledger C06/C08; KEY EVIDENCE.
- SOURCE_PROVENANCE_PIN = SRC-LD07-BISBEE-SYNTHETIC-REPLACEMENTS-2024--v1.0 — basis: Issue #5 proof rendering (Architecture Proof Protocol v0.1 AP-08 / Amendment v1.2 §6): explicit version-pinned public provenance of this ClaimVersion — the pinned field-6 Source record "Bisbee et al. Political Analysis 2024"; Drive-derived identity authority (Issue #39 Source inventory): Batch 02 §2; ledger C06; KEY EVIDENCE.
- SOURCE_PROVENANCE_PIN = SRC-LD07-XIE-SSDATABENCH-2026--v1.0 — basis: Issue #5 proof rendering (Architecture Proof Protocol v0.1 AP-08 / Amendment v1.2 §6): explicit version-pinned public provenance of this ClaimVersion — the pinned field-6 Source record "Xie et al. PNAS 2026"; Drive-derived identity authority (Issue #39 Source inventory): Batch 02 §3; ledger C06; KEY EVIDENCE.
- SOURCE_PROVENANCE_PIN = SRC-R03-WANG-MORGENSTERN-DICKERSON-NMI-2025--v1.0 — basis: Issue #5 proof rendering (Architecture Proof Protocol v0.1 AP-08 / Amendment v1.2 §6): explicit version-pinned public provenance of this ClaimVersion — the pinned field-6 Source record "Wang et al. NMI 2025"; Central HQ ruling embedded in this issue, resolution ruling §1.
- SOURCE_PROVENANCE_PIN = SRC-LD07-GONG-HUANG-CGSS-2025--v1.0 — basis: Issue #5 proof rendering (Architecture Proof Protocol v0.1 AP-08 / Amendment v1.2 §6): explicit version-pinned public provenance of this ClaimVersion — the pinned field-6 Source record "Gong & Huang 2025"; Drive-derived identity authority (Issue #39 Source inventory): Batch 02 §4; ledger C06/C08; KEY EVIDENCE.
- SOURCE_PROVENANCE_PIN = SRC-LD07-LING-CUI-SILICON-SAMPLES-2025--v1.0 — basis: Issue #5 proof rendering (Architecture Proof Protocol v0.1 AP-08 / Amendment v1.2 §6): explicit version-pinned public provenance of this ClaimVersion — the pinned field-6 Source record "Ling & Cui 2025"; Drive-derived identity authority (Issue #39 Source inventory): Batch 02 §5 (distinct study); ledger C06/C08; KEY EVIDENCE.