Can synthetic respondents recover strong treatment-effect heterogeneity?

Research question

In probability-sample survey experiments fielded after model training cutoffs and selected for substantial, well-powered moderation, do synthetic samples recover the sign and ranking of conditional treatment effects better than a model that assumes no heterogeneity?

Why it matters

The most-cited positive evidence shows LLMs forecast average effects well. Policy and theory usually turn on who responds differently, and current tests of interactions have little power because few archived experiments show strong moderation.

The debate lists this among what we still do not know: Can synthetic populations ever recover subgroup treatment effects and causal mechanisms out of distribution?

Evidence that motivates it

1. LLM simulations can usefully forecast the relative pattern of treatment effects in some US survey-experiment archives, while systematically overstating effect magnitudes. (LD07-HQ-C05 v1.1)
   Forecasting the pattern of average effects works in some US archives but overstates magnitudes.
   Cites: Ashokkumar et al. (2026), “Large language models can…”
2. Synthetic respondents can match selected survey averages while failing to reproduce real-world variance, regression relationships, individual structure and subgroup heterogeneity. (LD07-HQ-C06 v1.0)
   Synthetic samples can match selected averages while failing to reproduce variance and subgroup heterogeneity.
   Cites: Bisbee et al. (2024), “Synthetic Replacements for Human…”; Xie et al. (2026), “Evaluating the statistical realism of…”; Wang et al. NMI 2025; Gong Weigang & Huang Siyuan (2025), “整体事实还是有偏样本——基于大语言模型生成…”; Ling Wanying & Cui Sizhan (2025), “复现的限度——‘硅基样本’的可能与可为”
3. Statistical realism of synthetic responses is not a reliable proxy for treatment-effect accuracy. (LD07-HQ-C07 v1.0)
   Response-level realism does not predict treatment-effect accuracy, so CATE recovery must be tested directly.
   Cites: Li & Ji, “Statistical realism is not evidence that LLMs…”

Claims are shown as they read on the day this pack was built; claim pages keep every earlier version.

A possible design

Assemble post-cutoff experiments with pre-registered or published strong moderation (by partisanship, education, age, intersectional cells), including some outside the US. Generate persona-conditioned synthetic samples with several models. Compare estimated conditional effects with human estimates, using a no-heterogeneity benchmark and an honest power analysis, and report where synthetic samples beat, match or fall below that benchmark.

Data needed

Archives of recent survey experiments with microdata (for example TESS and journal replication files); post-2025 studies to avoid training contamination; API access to two or three models.

Feasibility

One PhD chapter or paper. Low: existing archives and modest API spend. The hard part is assembling enough post-cutoff experiments with strong moderation.

Has this been done?

Partly done already: related work covers some of this question (first-pass screen of 12 October 2026, checked again on 13 October 2026).

Ashokkumar et al. already test subgroup average effects (high accuracy) and pooled interactions (weak), but their ground truth contains few strongly moderated effects. Pew Data Labs (30 September 2026, after the evidence cut) found biased subgroup estimates and large cross-model divergence for descriptive polling. A 2026 preprint notes that moderation patterns vary across models in a single survey experiment, without a formal test. Recovery of strong, well-powered heterogeneity in post-cutoff experiments against a no-heterogeneity benchmark remains untested.

Literature check. Six new queries (preprint, journal, forward and plain web) found no study that selects post-cutoff survey experiments with strong, well-powered moderation and tests synthetic samples against a no-heterogeneity benchmark. Ashokkumar et al. (Nature 2026) and Li & Ji (arXiv 2026) remain the closest. New preprints touch moderation only in passing: Hoq & Weninger (arXiv:2604.15329) note that moderation patterns vary across models in one survey experiment without a formal heterogeneity test, and a persona-panel study (arXiv:2608.00979) reports only marginal contrasts. Checked by Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert.

Where the search could not reach. Screened by web search on the date shown; Google Scholar, SSRN and paywalled journal pages were only partly reachable. Absence of a hit is not proof that the work does not exist. Literature check of 2026-10-13: Full texts of the new preprints were not read (abstracts only). Forward citations of the Nature paper could not be listed with web search; only search-visible follow-ups were checked. The Pew Data Labs and Peng et al. entries were not re-checked. Absence of a hit is not proof that the work does not exist.

Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.

Nearest work found

• Ashokkumar, Hewitt, Ghezae & Willer, "Large language models can predict the results of social science experiments," Nature 656:115–122 (2026)
  Subgroup average effects predicted well; interaction effects predicted weakly, with only 6–15% of original effects significantly moderated.
• Chapekis, Lau, Bestvater, Shah, Mercer & Smith, "How synthetic polling results change based on the AI model," Pew Research Center Data Labs (30 Sep 2026)
  Digital twins of about 6,700 panelists err by 11–13 points on 2026 survey waves, with biased subgroup estimates and large divergence between models; descriptive items, not experiments. Published after LD-07's evidence cut.
• Li & Ji, "Statistical realism is not evidence that LLMs can estimate treatment effects in social science experiments," arXiv:2604.02458 (2026)
  Realism weakly related to treatment-effect accuracy across 62 countries; conditional effects are not the focus.
• Peng, Gui, Brucks, … Toubia, "Digital Twins as Funhouse Mirrors: Five Key Distortions," arXiv:2509.19088 (2025; v5 2026)
  Across 19 pre-registered studies, individual digital twins built from 500+ answers beat a generic LLM only slightly and show stereotyping, representation and ideological distortions; individual prediction, not conditional effects in experiments.
• Hoq & Weninger, "Evaluating LLMs as Human Surrogates in Controlled Experiments," arXiv:2604.15329 (v1, 8 March 2026) Judged from an abstract, summary or search listing only
  Abstract read: compares LLM and human responses in a survey experiment and reports that effect magnitudes and moderation patterns vary across models. One experiment; the abstract describes no formal heterogeneity test, no no-heterogeneity benchmark and no post-cutoff selection.
• "Passing Coarse Marginal Checks Can Be Cheap: Persona Mixtures and Imprecise Treatment-Response Estimates in an LLM Persona Panel," arXiv:2608.00979 (2026; authors not confirmed) Judged from an abstract, summary or search listing only Some details not confirmed
  Abstract read: one fixed model-prompt persona panel passes registered marginal criteria without precisely estimating the treatment-response object; no human comparison of subgroup or moderator effects. Supports the need for a stricter heterogeneity test.
• Sen, Ahnert, von der Heyde, Lasser, Weiß & Strohmaier, "Total Simulated Survey Error: Designing and Diagnosing Survey Responses from Large Language Models," arXiv:2609.10280 (9 September 2026) Judged from an abstract, summary or search listing only
  Abstract read: error framework for LLM survey simulation with case studies; the abstract does not mention experiments or treatment heterogeneity. Methodological context only.

References

Sources behind the motivating claims:

• Ashokkumar, Hewitt, Ghezae & Willer, “Large language models can predict the results of social science experiments,” Nature 656:115–122 (2026) (SRC-LD07-ASHOKKUMAR-NATURE-2026, original)
• Bisbee, Clinton, Dorff, Kenkel & Larson, “Synthetic Replacements for Human Survey Data? The Perils of Large Language Models,” Political Analysis 32(4):401–416 (2024) (SRC-LD07-BISBEE-SYNTHETIC-REPLACEMENTS-2024, original)
• Xie et al., “Evaluating the statistical realism of LLM-generated social science data,” PNAS 123(19):e2538145123 (2026) (SRC-LD07-XIE-SSDATABENCH-2026, original)
• Wang et al. NMI 2025 (SRC-R03-WANG-MORGENSTERN-DICKERSON-NMI-2025, no checked link to the original yet)
• Gong Weigang & Huang Siyuan, “整体事实还是有偏样本——基于大语言模型生成数据的测量,” Academic Monthly 57(6):123–136 (2025) (SRC-LD07-GONG-HUANG-CGSS-2025, original)
• Ling Wanying & Cui Sizhan, “复现的限度——‘硅基样本’的可能与可为,” Journal of Intelligent Society 4(2):179–201 (2025) (SRC-LD07-LING-CUI-SILICON-SAMPLES-2025, no checked link to the original yet)
• Li & Ji, “Statistical realism is not evidence that LLMs can estimate treatment effects in social science experiments,” arXiv:2604.02458 (SRC-LD07-LI-JI-REALISM-ATE-2026, original)

BibTeX

The first entry cites this opportunity; then its nearest work; then the sources above. Nearest work and sources are recorded as reference strings, so each goes in whole as the title; add authors and year from the original before you cite it.

@misc{sociolens_ro_ld07_04_v2,
  author = {{SocioLens project}},
  title = {Can synthetic respondents recover strong treatment-effect heterogeneity?},
  year = {2026},
  date = {2026-10-13},
  howpublished = {\url{https://sociolens-web.pages.dev/opportunities/ro-ld07-04/v2/}},
  note = {Research opportunity RO-LD07-04, version v2}
}

@misc{ro_ld07_04_nearest_1,
  title = {Ashokkumar, Hewitt, Ghezae \& Willer, "Large language models can predict the results of social science experiments," Nature 656:115–122 (2026)},
  url = {https://www.nature.com/articles/s41586-026-10742-x},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_04_nearest_2,
  title = {Chapekis, Lau, Bestvater, Shah, Mercer \& Smith, "How synthetic polling results change based on the AI model," Pew Research Center Data Labs (30 Sep 2026)},
  url = {https://www.pewresearch.org/data-labs/2026/09/30/how-synthetic-polling-results-change-based-on-the-ai-model},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_04_nearest_3,
  title = {Li \& Ji, "Statistical realism is not evidence that LLMs can estimate treatment effects in social science experiments," arXiv:2604.02458 (2026)},
  url = {https://arxiv.org/abs/2604.02458},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_04_nearest_4,
  title = {Peng, Gui, Brucks, … Toubia, "Digital Twins as Funhouse Mirrors: Five Key Distortions," arXiv:2509.19088 (2025; v5 2026)},
  url = {https://arxiv.org/abs/2509.19088},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_04_nearest_5,
  title = {Hoq \& Weninger, "Evaluating LLMs as Human Surrogates in Controlled Experiments," arXiv:2604.15329 (v1, 8 March 2026)},
  url = {https://arxiv.org/abs/2604.15329},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{ro_ld07_04_nearest_6,
  title = {"Passing Coarse Marginal Checks Can Be Cheap: Persona Mixtures and Imprecise Treatment-Response Estimates in an LLM Persona Panel," arXiv:2608.00979 (2026; authors not confirmed)},
  url = {https://arxiv.org/abs/2608.00979},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only; Some details not confirmed}
}

@misc{ro_ld07_04_nearest_7,
  title = {Sen, Ahnert, von der Heyde, Lasser, Weiß \& Strohmaier, "Total Simulated Survey Error: Designing and Diagnosing Survey Responses from Large Language Models," arXiv:2609.10280 (9 September 2026)},
  url = {https://arxiv.org/abs/2609.10280},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{src_ld07_ashokkumar_nature_2026,
  title = {Ashokkumar, Hewitt, Ghezae \& Willer, “Large language models can predict the results of social science experiments,” Nature 656:115–122 (2026)},
  url = {https://doi.org/10.1038/s41586-026-10742-x},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_ld07_bisbee_synthetic_replacements_2024,
  title = {Bisbee, Clinton, Dorff, Kenkel \& Larson, “Synthetic Replacements for Human Survey Data? The Perils of Large Language Models,” Political Analysis 32(4):401–416 (2024)},
  url = {https://doi.org/10.1017/pan.2024.5},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_ld07_xie_ssdatabench_2026,
  title = {Xie et al., “Evaluating the statistical realism of LLM-generated social science data,” PNAS 123(19):e2538145123 (2026)},
  url = {https://doi.org/10.1073/pnas.2538145123},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_r03_wang_morgenstern_dickerson_nmi_2025,
  title = {Wang et al. NMI 2025},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_ld07_gong_huang_cgss_2025,
  title = {Gong Weigang \& Huang Siyuan, “整体事实还是有偏样本——基于大语言模型生成数据的测量,” Academic Monthly 57(6):123–136 (2025)},
  url = {https://www.xsyk021.com/article/id/4c1886eb-c385-4019-92ff-d1cd75547fa5},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_ld07_ling_cui_silicon_samples_2025,
  title = {Ling Wanying \& Cui Sizhan, “复现的限度——‘硅基样本’的可能与可为,” Journal of Intelligent Society 4(2):179–201 (2025)},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_ld07_li_ji_realism_ate_2026,
  title = {Li \& Ji, “Statistical realism is not evidence that LLMs can estimate treatment effects in social science experiments,” arXiv:2604.02458},
  url = {https://arxiv.org/abs/2604.02458},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}