Do AI literature-review tools change the conclusions of social-science reviews?

Research question

Across a sample of published social-science meta-analyses, do the studies retrieved by AI review tools, compared with those the original review included, change the pooled effect, its heterogeneity or the stated conclusion, and is the missed evidence systematically different (grey literature, non-English studies, null results)?

Why it matters

Students and researchers now start literature reviews with AI search and deep-research tools. If those tools systematically miss certain kinds of evidence, they can shift what a field believes without anyone noticing, which matters most for the early-career researchers who rely on them.

The debate lists this among what we still do not know: How often does retrieval-augmented synthesis change which social-science evidence enters a review and therefore changes its conclusion?

Evidence that motivates it

1. High annotation accuracy does not by itself guarantee valid downstream inference; treating imperfect predicted labels as observed variables can bias estimates and invalidate confidence intervals, while design-based validation with a probability-sampled human-labelled subset can restore valid inference under stated assumptions. (LD07-HQ-C01 v1.0)
   The debate's core rule: validate the scientific conclusion, not just the intermediate output. For review tools the intermediate output is recall; the conclusion is the pooled finding.
   Cites: Egami et al. (2023), “Using Imperfect Surrogates for…”

Claims are shown as they read on the day this pack was built; claim pages keep every earlier version.

A possible design

Sample 20–30 recent social-science meta-analyses with open data (education, psychology, political science, sociology). Re-run each search question through several AI review tools. Recompute the pooled estimate using only the retrieved studies, and code what was missed by publication type, language and result direction.

Data needed

Meta-analyses with open study-level data; access to AI review tools (several have free tiers); a coding protocol for missed studies.

Feasibility

One PhD chapter or paper. Low: free or entry-level tool tiers and existing meta-analytic data. Tool behaviour changes quickly, so record versions and dates.

Has this been done?

Partly done already: related work covers some of this question (first-pass screen of 12 October 2026, checked again on 13 October 2026).

Medical and health studies measure AI-tool recall (Elicit sensitivity around 38% across four health reviews), a deep-research audit shows search skews evidence direction on a systematic-review benchmark, and one 2026 agentic pipeline re-ran a single psychology/communication meta-analysis and turned a null pooled effect significant. No study found re-runs a sample of published social-science meta-analyses through several AI review tools, tests how often the pooled conclusion changes, and codes whether the missed evidence is systematically grey, non-English or null.

Literature check. The closest new work is Taherinezhad et al. (AutoSynthesis, arXiv:2607.15247), which re-runs one published psychology/communication meta-analysis with an agentic pipeline and finds the pooled effect turns significant, and Lau et al. (Cochrane ESM 2025), which measures Elicit recall against four health reviews without testing conclusions. Neither samples a set of social-science meta-analyses, compares several review tools, or codes the missed studies by type, language or result direction, so the core design is still open. Checked by Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert.

Where the search could not reach. Screened by web search on the date shown; Google Scholar, SSRN and paywalled journal pages were only partly reachable. Absence of a hit is not proof that the work does not exist. Literature check of 2026-10-13: Wiley (Cochrane ESM), PMC, ScienceDirect and Medscape pages were blocked or failed to load; a 2026 ScienceDirect paper titled 'Evaluating the Utility of Artificial Intelligence in Conducting Systematic Reviews' (S2352344126000841) and a 2026 Medscape report 'AI Falls Short in Systematic Review Searches' could not be opened and should be checked by hand. Scholarly APIs and Google Scholar not used.

Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.

Nearest work found

• Xiao, Wang, Chen et al., "Search Shapes Conclusions: Auditing Evidence Selection Bias in Deep Research Agents," arXiv:2609.39026 (2026)
  Adaptive search by deep-research agents skews evidence direction and shifts estimates; biomedical benchmark, not published social-science reviews.
• Bernard et al., "Using artificial intelligence for systematic review: the example of elicit," BMC Medical Research Methodology 25:75 (2025)
  Elicit retrieved only 3 of the umbrella review's included studies (missing 17) plus 3 new ones, with results varying across repeated runs; recall only.
• Hilkenmeier et al., "Evaluating the AI Tool 'Elicit' as a Semi-Automated Second Reviewer for Data Extraction," Social Science Computer Review (2025)
  Tests data extraction accuracy, not search or conclusions.
• Taherinezhad, Maier, Vitagliano, Pierri & Feuerriegel, "AutoSynthesis: An agentic system for automated meta-analysis," arXiv:2607.15247 (2026)
  Opened. An LLM search agent (arXiv, Semantic Scholar, Crossref, OSF, PubMed) re-ran one published meta-analysis on the persuasiveness of LLM-generated messages: it recovered 5 of 7 originally included studies (6 of 7 after manual correction) and added others, and its pooled Hedges' g was 0.143 (significant) versus the original 0.020 (not significant). This is a single-case demonstration that AI retrieval can change a social-science pooled conclusion; it does not sample many meta-analyses, compare commercial review tools, or code missed studies by publication type, language or result direction.
• Lau et al., "Comparison of Elicit AI and Traditional Literature Searching in Evidence Syntheses Using Four Case Studies," Cochrane Evidence Synthesis and Methods (2025), doi:10.1002/cesm.70050; preprint medRxiv 10.1101/2025.06.17.25329772
  Opened the medRxiv preprint (the journal page and PMC copy were not reachable; first author taken from the Wiley search listing). Elicit sensitivity against four health reviews (vaping harms, physical-activity economics, fertility, breast reconstruction) averaged 37.9%. It counts unique extra studies but does not test whether conclusions or pooled estimates change; health topics only.
• Xie, Su, Zhou, Liu, Zhang & Ai, "MetaSyn: A Benchmark for LLM Agents on Meta-Analysis Articles from Nature Portfolio," arXiv:2606.17041 (2026) Judged from an abstract, summary or search listing only
  Opened abstract page. 422 meta-analyses from Nature Portfolio with included studies and a PubMed-anchored corpus with distractors; evaluates agents stage by stage. The abstract does not describe live search with real tools, recomputing pooled effects, or a social-science subset, so it is a benchmark resource rather than a test of conclusion change.
• Zabaleta & Lin, "SciLitBench: Benchmark and Design Principles for LLM-Powered Systematic Literature Reviews," arXiv:2609.05505 (2026)
  Opened. Evaluates 22 open-weight LLMs on screening and extraction for one review about literature-review automation; no search re-runs of published meta-analyses and no pooled estimates.
• Lagisz, Mizuno, Morrison, Pollo, Ricolfi, Yang & Nakagawa, "Using Elicit AI research assistant for data extraction in systematic reviews: a feasibility study across environmental and life sciences," Research Synthesis Methods (2025)
  Opened the EcoEvoRxiv preprint. Data extraction only, environmental and life sciences; explicitly did not extract effect sizes or test search.

References

Sources behind the motivating claims:

• Egami, Hinck, Stewart & Wei, “Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Models,” NeurIPS 2023 (SRC-LD07-EGAMI-DOWNSTREAM-INFERENCE-2023, original)

BibTeX

The first entry cites this opportunity; then its nearest work; then the sources above. Nearest work and sources are recorded as reference strings, so each goes in whole as the title; add authors and year from the original before you cite it.

@misc{sociolens_ro_ld07_06_v2,
  author = {{SocioLens project}},
  title = {Do AI literature-review tools change the conclusions of social-science reviews?},
  year = {2026},
  date = {2026-10-13},
  howpublished = {\url{https://sociolens-web.pages.dev/opportunities/ro-ld07-06/v2/}},
  note = {Research opportunity RO-LD07-06, version v2}
}

@misc{ro_ld07_06_nearest_1,
  title = {Xiao, Wang, Chen et al., "Search Shapes Conclusions: Auditing Evidence Selection Bias in Deep Research Agents," arXiv:2609.39026 (2026)},
  url = {https://arxiv.org/abs/2609.39026},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_06_nearest_2,
  title = {Bernard et al., "Using artificial intelligence for systematic review: the example of elicit," BMC Medical Research Methodology 25:75 (2025)},
  url = {https://link.springer.com/article/10.1186/s12874-025-02528-y},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_06_nearest_3,
  title = {Hilkenmeier et al., "Evaluating the AI Tool 'Elicit' as a Semi-Automated Second Reviewer for Data Extraction," Social Science Computer Review (2025)},
  url = {https://journals.sagepub.com/doi/10.1177/08944393251404052},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_06_nearest_4,
  title = {Taherinezhad, Maier, Vitagliano, Pierri \& Feuerriegel, "AutoSynthesis: An agentic system for automated meta-analysis," arXiv:2607.15247 (2026)},
  url = {https://arxiv.org/abs/2607.15247},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_06_nearest_5,
  title = {Lau et al., "Comparison of Elicit AI and Traditional Literature Searching in Evidence Syntheses Using Four Case Studies," Cochrane Evidence Synthesis and Methods (2025), doi:10.1002/cesm.70050; preprint medRxiv 10.1101/2025.06.17.25329772},
  url = {https://onlinelibrary.wiley.com/doi/full/10.1002/cesm.70050},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_06_nearest_6,
  title = {Xie, Su, Zhou, Liu, Zhang \& Ai, "MetaSyn: A Benchmark for LLM Agents on Meta-Analysis Articles from Nature Portfolio," arXiv:2606.17041 (2026)},
  url = {https://arxiv.org/abs/2606.17041},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{ro_ld07_06_nearest_7,
  title = {Zabaleta \& Lin, "SciLitBench: Benchmark and Design Principles for LLM-Powered Systematic Literature Reviews," arXiv:2609.05505 (2026)},
  url = {https://arxiv.org/abs/2609.05505},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_06_nearest_8,
  title = {Lagisz, Mizuno, Morrison, Pollo, Ricolfi, Yang \& Nakagawa, "Using Elicit AI research assistant for data extraction in systematic reviews: a feasibility study across environmental and life sciences," Research Synthesis Methods (2025)},
  url = {https://www.cambridge.org/core/journals/research-synthesis-methods/article/using-elicit-ai-research-assistant-for-data-extraction-in-systematic-reviews-a-feasibility-study-across-environmental-and-life-sciences/C97DAEC70C3173A260F0B12E729E7250},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_ld07_egami_downstream_inference_2023,
  title = {Egami, Hinck, Stewart \& Wei, “Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Models,” NeurIPS 2023},
  url = {https://doi.org/10.52202/075280-3000},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}