Research opportunity · AI in the research apparatus
Does quantization or the serving engine change social-science findings?
In short: has this been done?
Related work: No study found. A first-pass search found no study that does the core design. It has not been checked in depth, so a fuller search could still find one.
How far checked: First-pass screen only. A first-pass AI web search on 12 October 2026. No deeper check yet. Every search and check so far was done by AI; no person or domain expert has checked them yet.
The searches, the nearest work found and what the search could not reach
Earlier version (v1)
This is RO-LD07-02 as it was published at v1, kept unchanged. The current version is v2; its changes are listed there.
Question
With weights, prompts and hardware held fixed, do fp16, 8-bit and 4-bit builds of open-weight models, and different inference engines, change substantive estimates on published annotation and synthetic-respondent tasks more than prompt paraphrasing does?
Gap type: Instrument stabilityMaster's thesisLow costRelated work: No study found
Why it matters
Open-weight models are the main route to reproducible, low-cost LLM research, and most researchers run quantized builds on whatever engine is available. If these invisible choices move conclusions, methods sections need to report them and reviewers need to ask.
The debate lists this among what we still do not know: For open-weight models, do deployment choices such as quantization materially change social-science reliability metrics?
Evidence that motivates it
- In a large 2025 preprint spanning 37 annotation tasks from 21 published studies, plausible choices of LLM and annotation configuration produced incorrect inferential conclusions in about one third of tested hypotheses for state-of-the-art models, with higher rates for smaller models and near conventional significance thresholds. (LD07-HQ-C02 v1.0)
How this study uses it: Analytic flexibility in model configuration already changes inferential conclusions; deployment settings are an unmeasured extension of that configuration space. - Model identity is time-indexed: proprietary services can change materially across deployed snapshots, so exact version and date are part of the measurement protocol. (LD07-HQ-C12 v1.0)
How this study uses it: The instrument is defined by more than its name; deployment is part of its identity.
Each claim is shown as it read when this version was published.
A possible design
Take 10–15 annotation and silicon-sample tasks with published estimates. Run two or three open-weight model families at several precisions (fp16, 8-bit, 4-bit in common formats) and on two or three serving engines, holding prompts and seeds fixed. Compare the spread of substantive estimates across deployment settings with the spread across prompt paraphrases, using the paraphrase spread as the benchmark for "material".
Data needed
Public replication datasets; open-weight model checkpoints and their quantized builds; one or two consumer or rented GPUs.
Feasibility
Master's thesis. Very low: open-weight models only, a few days of GPU time. A strong first project for a computational methods student.
Has this been done?
The search found no study that does the core design (first-pass screen of 12 October 2026; not yet checked in depth).
Work exists on inference-backend effects on ML benchmarks and on decoding and sampling randomness in research use, but none found varies precision and serving engine and measures changes in social-science conclusions.
How strong this evidence is
- Who searched and checked: an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: none of the 3 nearest works below is recorded as judged from an abstract only. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search on the date shown; Google Scholar, SSRN and paywalled journal pages were only partly reachable. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- Pape, Evertz & Schönherr, "The Silent Hyperparameter," arXiv:2605.19537 (2026)
Inference backend alone shifts ML benchmark scores by up to 16.6 points; no social-science tasks, no quantization. - Singh & Sajjad, "Interpreting the Effects of Quantization on LLMs," IJCNLP-AACL 2025, arXiv:2508.16785
Finds minor effects of 4- and 8-bit quantization on calibration and internals; no task-level downstream conclusions. - Cummins, "The threat of analytic flexibility in using LLMs to simulate human data," arXiv:2509.13397 (2025; v4 2026)
Silicon-sample fidelity varies widely across 252 model, sampling and prompt configurations; does not cover quantization or engines.
Search queries used
quantization effects LLM annotation social science 4-bit 8-bit text classification validityinference engine vLLM llama.cpp nondeterminism hardware LLM outputs differ survey simulation or annotation reproducibility social sciencequantized open-weight models silicon sampling public opinion survey responses quantization biasLLM hacking researcher degrees of freedom model configuration temperature annotation statistical conclusions
Cite this page
SocioLens project. (2026, October 12). Does quantization or the serving engine change social-science findings? [Research opportunity RO-LD07-02, version v1]. SocioLens. https://sociolens-web.pages.dev/opportunities/ro-ld07-02/v1/
@misc{sociolens_ro_ld07_02_v1,
author = {{SocioLens project}},
title = {Does quantization or the serving engine change social-science findings?},
year = {2026},
date = {2026-10-12},
howpublished = {\url{https://sociolens-web.pages.dev/opportunities/ro-ld07-02/v1/}},
note = {Research opportunity RO-LD07-02, version v1}
}The link points to this version, whose record never changes. For a finding, cite the original study as well: SocioLens summarizes research and does not replace it.
Provenance
RO-LD07-02 v1. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-07 debate on 2026-10-12. Checked by an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.