Do LLM judges inflate the quality of AI-written deliberation?

Research question

On a corpus mixing human-written and LLM-written deliberative contributions, do LLM judges assign higher construct scores (for example on the Discourse Quality Index or argument quality) to LLM-authored text than expert coders blind to authorship, and is the gap larger when the judge model also wrote the text?

Why it matters

LLM judges are moving into social science to score deliberation, civility and argument quality, at the same time as online discussion fills with AI-written text. A judge that favours machine prose would quietly bias measures of public deliberation.

Evidence that motivates it

1. LLM judges can correlate with human evaluations while exhibiting systematic evaluator-specific biases, including position bias and self-preference; task-specific validation is therefore required. (LD07-HQ-C13 v1.0)
   LLM judges show position bias and self-preference, so construct-level validation is required.
   Cites: Shi et al. (2025), “Judging the Judges: A Systematic…”; Panickssery et al. (2024), “LLM Evaluators Recognize and…”; Calderon et al. (2025), “The Alternative Annotator Test…”
2. Whether an LLM can replace human annotation is a task-specific empirical proposition that can sometimes be justified with an explicit statistical replacement test; it is not a general property of LLMs. (LD07-HQ-C03 v1.0)
   Replacing human judgement is a task-specific claim that has to be tested, not assumed.
   Cites: Calderon et al. (2025), “The Alternative Annotator Test…”

Claims are shown as they read on the day this pack was built; claim pages keep every earlier version.

A possible design

Build a corpus of matched deliberative contributions on the same prompts written by people and by several LLMs. Have expert coders score them with an established instrument while blind to authorship. Score the same items with several LLM judges, including each generating model. Estimate the authorship gap between judges and experts, and the extra gap for self-authored text.

Data needed

Existing deliberation corpora (for example from online deliberation platforms or citizens' assemblies); generated counterparts; expert coder time.

Feasibility

One PhD chapter or paper. Low to medium: coder time is the main cost.

Has this been done?

Partly done already: related work covers some of this question (first-pass screen of 12 October 2026, checked again on 13 October 2026).

LLM scoring of deliberation quality has been validated on human-written text (Gelauff et al.), and LLMs prefer LLM-written product, paper and film descriptions over human judges' choices (Laurito et al.), though a reanalysis finds a shared taste for model-written text rather than a preference for a model's own text, and blind evaluation removes self-preference in another task (Chae et al.). No study found tests judge-mode scoring of an established deliberation construct on mixed human and LLM text against authorship-blind experts, or estimates the extra gap for self-authored deliberative text.

Literature check. Six new queries found no study that scores an established deliberation construct (DQI, argument quality) on a mixed human and LLM corpus with LLM judges against authorship-blind expert coders. New work sharpens rather than settles the question: a reanalysis of the Laurito et al. data finds a shared preference for model-written text but no own-model premium (Żatuchin, Sept 2026), a blind-evaluation study finds self-preference vanishes when authorship labels are removed but labels alone shift scores (Chae et al., 2026), and a deliberation study compares LLM agent groups with citizen assemblies on the Deliberative Reason Index at group level without testing judge bias (Flechtner, Aug 2026). None of these uses deliberative text with expert coders as the benchmark. Checked by Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert.

Where the search could not reach. Screened by web search on the date shown; Google Scholar, SSRN and paywalled journal pages were only partly reachable. Absence of a hit is not proof that the work does not exist. Literature check of 2026-10-13: Six searches were run for this opportunity under the shared budget; full texts of Chae et al. and Flechtner were not read, so whether either uses human scorers is unconfirmed from abstracts. Google Scholar and political-communication journals beyond those listed were not searched. Absence of a hit is not proof that the work does not exist.

Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.

Nearest work found

• Gelauff, Goyal, Dindukurthi, Goel & Siu, "Estimating Contribution Quality in Online Deliberations Using a Large Language Model," arXiv:2408.11936 (2024)
  LLM scores deliberation contributions well against human averages; all texts human-written.
• Laurito, Davis, Grietzer, Gavenčiak, Böhm & Kulveit, "AI–AI bias: large language models favor communications generated by large language models," PNAS 122 (2025)
  LLMs prefer LLM-written descriptions of products, papers and films; not social-science constructs.
• Chehbouni et al., "Neither Valid nor Reliable? Investigating the Use of LLMs as Judges," arXiv:2508.18076 (2025)
  Conceptual critique using measurement theory; no empirical construct test.
• Żatuchin, "A Shared Taste for Model-Written Text: The Generator-by-Selector Matrices of 'AI-AI Bias' Show No Detectable Own-Model Premium," arXiv:2610.00369 (30 Sep 2026)
  Reanalyses the public Laurito et al. data: no own-model premium (pooled +0.019, p = 0.14; design could detect about 0.04 or more), but a robust shared preference for model-written text; contrasts with a sizeable own-model term in Tan et al. (ACL 2024) data. Product, abstract and film descriptions only; no new human judges and no deliberation construct. Directly bears on the opportunity's 'extra gap for self-authored text' sub-question.
• Chae, Kim, Jung, Choi & Jung, "Self- and Other-Labels Induce Bidirectional Bias in LLM Judges," arXiv:2608.18091 (v2, Sep 2026)
  Ten LLMs judge sets of narrative constraints; under blind evaluation self-preference disappears, while self/other authorship labels inflate or deflate scores regardless of true source. Not deliberative text and no human baseline mentioned in the abstract; shows the blind versus labelled condition matters for the proposed design.
• Chen, Wang, Zhang, Hu & Lin, "Beyond the Surface: Measuring Self-Preference in LLM Judgments," EMNLP 2025, pp. 1653-1672
  Proposes measuring self-preference against gold judgments of quality rather than raw own-versus-other scores, the same logic as the opportunity's expert benchmark. General NLP tasks; the abstract does not say the gold judgments are human, and there is no deliberation construct.
• Panickssery, Bowman & Feng, "LLM Evaluators Recognize and Favor Their Own Generations," NeurIPS 2024
  Authors and title checked on the arXiv page (arXiv:2404.13076); the NeurIPS 2024 venue is from search results. Summarisation tasks: self-recognition correlates with self-preference. Not social-science constructs or expert coders.
• Flechtner, "The Deliberative Deficit: An Empirical Critique of LLMs in Democratic Discourse," arXiv:2608.10186 (10 Aug 2026)
  1,980 runs of five-agent LLM groups on 12 citizen-assembly topics scored with the Deliberative Reason Index and compared with human assemblies at group level. The abstract does not say who scored the text or test whether LLM judges inflate AI-written contributions.

References

Sources behind the motivating claims:

• Shi et al., “Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge,” IJCNLP-AACL 2025 (SRC-LD07-SHI-JUDGING-THE-JUDGES-2025, original)
• Panickssery, Bowman & Feng, “LLM Evaluators Recognize and Favor Their Own Generations,” NeurIPS 2024 (SRC-LD07-PANICKSSERY-SELF-PREFERENCE-2024, original)
• Calderon, Reichart & Dror, “The Alternative Annotator Test for LLM-as-a-Judge,” ACL 2025 (SRC-LD07-CALDERON-ALTERNATIVE-ANNOTATOR-2025, original)

BibTeX

The first entry cites this opportunity; then its nearest work; then the sources above. Nearest work and sources are recorded as reference strings, so each goes in whole as the title; add authors and year from the original before you cite it.

@misc{sociolens_ro_ld07_09_v2,
  author = {{SocioLens project}},
  title = {Do LLM judges inflate the quality of AI-written deliberation?},
  year = {2026},
  date = {2026-10-13},
  howpublished = {\url{https://sociolens-web.pages.dev/opportunities/ro-ld07-09/v2/}},
  note = {Research opportunity RO-LD07-09, version v2}
}

@misc{ro_ld07_09_nearest_1,
  title = {Gelauff, Goyal, Dindukurthi, Goel \& Siu, "Estimating Contribution Quality in Online Deliberations Using a Large Language Model," arXiv:2408.11936 (2024)},
  url = {https://arxiv.org/abs/2408.11936},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_09_nearest_2,
  title = {Laurito, Davis, Grietzer, Gavenčiak, Böhm \& Kulveit, "AI–AI bias: large language models favor communications generated by large language models," PNAS 122 (2025)},
  url = {https://arxiv.org/abs/2407.12856},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_09_nearest_3,
  title = {Chehbouni et al., "Neither Valid nor Reliable? Investigating the Use of LLMs as Judges," arXiv:2508.18076 (2025)},
  url = {https://arxiv.org/abs/2508.18076},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_09_nearest_4,
  title = {Żatuchin, "A Shared Taste for Model-Written Text: The Generator-by-Selector Matrices of 'AI-AI Bias' Show No Detectable Own-Model Premium," arXiv:2610.00369 (30 Sep 2026)},
  url = {https://arxiv.org/abs/2610.00369},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_09_nearest_5,
  title = {Chae, Kim, Jung, Choi \& Jung, "Self- and Other-Labels Induce Bidirectional Bias in LLM Judges," arXiv:2608.18091 (v2, Sep 2026)},
  url = {https://arxiv.org/abs/2608.18091},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_09_nearest_6,
  title = {Chen, Wang, Zhang, Hu \& Lin, "Beyond the Surface: Measuring Self-Preference in LLM Judgments," EMNLP 2025, pp. 1653-1672},
  url = {https://aclanthology.org/2025.emnlp-main.86/},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_09_nearest_7,
  title = {Panickssery, Bowman \& Feng, "LLM Evaluators Recognize and Favor Their Own Generations," NeurIPS 2024},
  url = {https://proceedings.neurips.cc/paper_files/paper/2024/hash/7f1f0218e45f5414c79c0679633e47bc-Abstract-Conference.html},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_09_nearest_8,
  title = {Flechtner, "The Deliberative Deficit: An Empirical Critique of LLMs in Democratic Discourse," arXiv:2608.10186 (10 Aug 2026)},
  url = {https://arxiv.org/abs/2608.10186},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_ld07_shi_judging_the_judges_2025,
  title = {Shi et al., “Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge,” IJCNLP-AACL 2025},
  url = {https://doi.org/10.18653/v1/2025.ijcnlp-long.18},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_ld07_panickssery_self_preference_2024,
  title = {Panickssery, Bowman \& Feng, “LLM Evaluators Recognize and Favor Their Own Generations,” NeurIPS 2024},
  url = {https://proceedings.neurips.cc/paper_files/paper/2024/hash/7f1f0218e45f5414c79c0679633e47bc-Abstract-Conference.html},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_ld07_calderon_alternative_annotator_2025,
  title = {Calderon, Reichart \& Dror, “The Alternative Annotator Test for LLM-as-a-Judge,” ACL 2025},
  url = {https://doi.org/10.18653/v1/2025.acl-long.782},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}