Do pinned model snapshots keep social-science conclusions stable over time?

Research question

On a fixed battery of published social-science annotation tasks, how often does a substantive conclusion (sign, significance, interval coverage) change between repeated queries of a pinned model snapshot, a moving model alias over 6–12 months, and successive open-weight releases, and can design-based correction absorb those changes?

Why it matters

Researchers increasingly cite a model name as if it identified a fixed instrument. If conclusions drift with silent updates or release lines, published LLM-annotated findings may not be reproducible even with the same code and prompt.

The debate lists this among what we still do not know: How stable are 2025–2026 model snapshots over time when used for the same social-science measurement task?

Evidence that motivates it

1. Model identity is time-indexed: proprietary services can change materially across deployed snapshots, so exact version and date are part of the measurement protocol. (LD07-HQ-C12 v1.0)
   Model identity is time-indexed; exact version and date belong in the protocol.
   Cites: Chen et al. (2024), “How is ChatGPT’s behavior changing…”
2. In a large 2025 preprint spanning 37 annotation tasks from 21 published studies, plausible choices of LLM and annotation configuration produced incorrect inferential conclusions in about one third of tested hypotheses for state-of-the-art models, with higher rates for smaller models and near conventional significance thresholds. (LD07-HQ-C02 v1.0)
   Configuration choices already flip about a third of tested conclusions at a single point in time.
   Cites: Baumann et al., “Large Language Model Hacking”
3. High annotation accuracy does not by itself guarantee valid downstream inference; treating imperfect predicted labels as observed variables can bias estimates and invalidate confidence intervals, while design-based validation with a probability-sampled human-labelled subset can restore valid inference under stated assumptions. (LD07-HQ-C01 v1.0)
   Design-based correction is the candidate remedy whose robustness to drift is untested.
   Cites: Egami et al. (2023), “Using Imperfect Surrogates for…”

Claims are shown as they read on the day this pack was built; claim pages keep every earlier version.

A possible design

Pre-register a battery of 10–20 replication tasks with public data and published estimates (for example from the Baumann et al. or Yang et al. task sets). Re-run them monthly for 6–12 months on pinned snapshot IDs, on the corresponding moving aliases, and on two or three open-weight release lines. The outcome is the substantive estimate, not label agreement. Then apply design-based supervised learning with a fixed human-labelled subsample and test whether it absorbs the drift.

Data needed

Published replication datasets and codebooks; API access to two providers; one GPU for open-weight runs; a human-labelled validation subsample per task (existing gold labels where available).

Feasibility

One PhD chapter or paper. Low: API spend in the low hundreds of US dollars plus modest GPU time. Needs time rather than money: the value comes from the longitudinal window.

Has this been done?

Partly done already: related work covers some of this question (first-pass screen of 12 October 2026, checked again on 13 October 2026).

Barrie, Palmer and Spirling already show month-to-month drift for closed models and fragile regression conclusions across model versions, Chen, Zaharia and Zou showed version-to-version behaviour change in 2023, and Tschisgale and Wulff (2026) found periodic variability in GPT-4o over three months on a non-social-science task. What remains open is a pre-registered comparison of pinned snapshots versus aliases versus open-weight release lines, with flips in substantive social-science estimates as the outcome and a test of whether design-based correction absorbs them.

Literature check. No study was found that pre-registers repeated runs of social-science annotation tasks over 6-12 months comparing pinned snapshots, moving aliases and open-weight release lines, with flips in substantive estimates as the outcome and a test of design-based correction. Barrie, Palmer and Spirling remain the closest work (monthly re-runs, closed models plus Llama-2), and Chen, Zaharia and Zou showed version-to-version behaviour change in 2023 on non-social-science tasks. New 2026 work (Tschisgale and Wulff on periodic GPT-4o variability; Abraham and Bucknall on disclosure of silent updates) strengthens the motivation but does not do the core design. Checked by Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert.

Where the search could not reach. Screened by web search on the date shown; Google Scholar, SSRN and paywalled journal pages were only partly reachable. Absence of a hit is not proof that the work does not exist. Literature check of 2026-10-13: The Barrie, Palmer and Spirling full paper and the Political Communication critique by Balluff et al. could not be read in full (only the slide deck and the ITU repository abstract). The Wiley article by Aksoy (2026) on political bias over time could not be opened; it appears to test political-bias benchmarks over GPT versions rather than annotation tasks, but this is unconfirmed. Scholarly APIs were not used. Absence of a hit is not proof that the work does not exist.

Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.

Nearest work found

• Barrie, Palmer & Spirling, "Replication for Language Models: Problems, Principles, and Best Practice for Political Science," working paper (Dec 2024 draft)
  Monthly re-runs of labelling tasks show variability in closed models (GPT-4 and Gemini, per the companion slides) and changed regression conclusions; includes one locally run open-weight model (Llama-2) as a stable comparator but does not track open-weight release lines or pinned snapshots.
• Fan, Li, Zhuang, Sun & Wang, "The Moving Target: A Longitudinal Audit of Trust-Benchmark Score Drift Across Open-Source Chat LLM Release Lines," arXiv:2607.02587 (2026)
  Tracks trust-benchmark drift across three generations of four open-weight families; no social-science tasks or downstream estimates.
• Coqueret et al., "Randomness in large language models: What researchers need to know (and report)," arXiv:2607.24372 (2026)
  Shows sampling randomness changes downstream regressions and names silent updates as a source; does not measure drift over time.
• Chen, Zaharia & Zou, "How is ChatGPT's behavior changing over time?," arXiv:2307.09009 (2023)
  Compares March and June 2023 releases of GPT-3.5 and GPT-4 on seven tasks (including opinion surveys) and finds substantial behaviour changes between versions. Two time points, no social-science annotation battery, no downstream estimates, no open-weight lines or correction. A version was published in Harvard Data Science Review 6(2) (2024), doi:10.1162/99608f92.5317da47 (checked against the DOI's Crossref record on 21 October 2026).
• Tschisgale & Wulff, "Daily and Weekly Periodicity in Large Language Model Performance and Its Implications for Research," arXiv:2602.15889 (2026)
  Queries GPT-4o every three hours for about three months on one physics task with fixed settings and finds periodic variability accounting for about 20% of variance. Shows time-indexed instability of a nominally fixed model; abstract does not say whether a dated snapshot was pinned; not a social-science task and no downstream estimates.
• Abraham & Bucknall, "Silent Updates: Measuring and Closing the Post-Deployment Disclosure Gap," arXiv:2608.11803 (2026)
  Scores post-deployment disclosure practices of nine API providers and seven inference hosts. Documents that model changes are often undisclosed; does not measure output drift or any research conclusion.
• Trisovic, "The Shrinking Lifespan of LLMs in Science," arXiv:2604.07530 (2026)
  Bibliometric study of 62 LLMs in 108,000 citing papers: model lifespans in science are shortening by about 23% per release year. Motivates the problem (models retired quickly) but does not re-run any task.

References

Sources behind the motivating claims:

• Chen, Zaharia & Zou, “How is ChatGPT’s behavior changing over time?,” Harvard Data Science Review 6(2) (2024) (SRC-LD07-CHEN-ZAHARIA-ZOU-DRIFT-2024, original)
• Baumann, Röttger, Urman, Wendsjö, Plaza-del-Arco, Gruber & Hovy, “Large Language Model Hacking,” arXiv:2509.08825v2 (SRC-LD07-BAUMANN-LLM-HACKING-2025, original)
• Egami, Hinck, Stewart & Wei, “Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Models,” NeurIPS 2023 (SRC-LD07-EGAMI-DOWNSTREAM-INFERENCE-2023, original)

BibTeX

The first entry cites this opportunity; then its nearest work; then the sources above. Nearest work and sources are recorded as reference strings, so each goes in whole as the title; add authors and year from the original before you cite it.

@misc{sociolens_ro_ld07_01_v3,
  author = {{SocioLens project}},
  title = {Do pinned model snapshots keep social-science conclusions stable over time?},
  year = {2026},
  date = {2026-10-21},
  howpublished = {\url{https://sociolens-web.pages.dev/opportunities/ro-ld07-01/v3/}},
  note = {Research opportunity RO-LD07-01, version v3}
}

@misc{ro_ld07_01_nearest_1,
  title = {Barrie, Palmer \& Spirling, "Replication for Language Models: Problems, Principles, and Best Practice for Political Science," working paper (Dec 2024 draft)},
  url = {https://arthurspirling.org/documents/BarriePalmerSpirling_TrustMeBro.pdf},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_01_nearest_2,
  title = {Fan, Li, Zhuang, Sun \& Wang, "The Moving Target: A Longitudinal Audit of Trust-Benchmark Score Drift Across Open-Source Chat LLM Release Lines," arXiv:2607.02587 (2026)},
  url = {https://arxiv.org/abs/2607.02587},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_01_nearest_3,
  title = {Coqueret et al., "Randomness in large language models: What researchers need to know (and report)," arXiv:2607.24372 (2026)},
  url = {https://arxiv.org/abs/2607.24372},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_01_nearest_4,
  title = {Chen, Zaharia \& Zou, "How is ChatGPT's behavior changing over time?," arXiv:2307.09009 (2023)},
  url = {https://arxiv.org/abs/2307.09009},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_01_nearest_5,
  title = {Tschisgale \& Wulff, "Daily and Weekly Periodicity in Large Language Model Performance and Its Implications for Research," arXiv:2602.15889 (2026)},
  url = {https://arxiv.org/abs/2602.15889},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_01_nearest_6,
  title = {Abraham \& Bucknall, "Silent Updates: Measuring and Closing the Post-Deployment Disclosure Gap," arXiv:2608.11803 (2026)},
  url = {https://arxiv.org/abs/2608.11803},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_01_nearest_7,
  title = {Trisovic, "The Shrinking Lifespan of LLMs in Science," arXiv:2604.07530 (2026)},
  url = {https://arxiv.org/abs/2604.07530},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_ld07_chen_zaharia_zou_drift_2024,
  title = {Chen, Zaharia \& Zou, “How is ChatGPT’s behavior changing over time?,” Harvard Data Science Review 6(2) (2024)},
  url = {https://doi.org/10.1162/99608f92.5317da47},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_ld07_baumann_llm_hacking_2025,
  title = {Baumann, Röttger, Urman, Wendsjö, Plaza-del-Arco, Gruber \& Hovy, “Large Language Model Hacking,” arXiv:2509.08825v2},
  url = {https://arxiv.org/abs/2509.08825v2},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_ld07_egami_downstream_inference_2023,
  title = {Egami, Hinck, Stewart \& Wei, “Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Models,” NeurIPS 2023},
  url = {https://doi.org/10.52202/075280-3000},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}