Research opportunity · What kind of thing is generative AI, and does the answer predict anything?
Do the failure patterns of 2023–2024 language models persist in reasoning models?
In short: has this been done?
Related work: Partly addressed. Related studies cover parts of this question; what they leave open is set out below. Only a first-pass search has been done, so it is not yet known whether a fuller search would find a study that does the whole design.
How far checked: First-pass screen only. A first-pass AI web search on 16 October 2026. No deeper check yet. Every search and check so far was done by AI; no person or domain expert has checked them yet.
The searches, the nearest work found and what the search could not reach
This page shows RO-LD01-02 version v1, whose record never changes. It is the current version; if the opportunity is revised, the newer version appears at its main page.
Question
When the same items that showed sensitivity to output probability, the reversal curse, irrelevant clauses, compositional depth and counterfactual premises are put to successive model generations from 2023 to 2026, including reasoning and agentic models, which failure patterns shrink in size but keep their shape, and which disappear?
Gap type: Untested beyond the original settingOne PhD chapter or paperNeeds a budget (medium or higher)Related work: Partly addressed
Why it matters
The model-level evidence in this debate comes mostly from 2023–2024 systems, and it is read both as mimicry and as a tool with known failure modes. If the patterns keep their shape in newer models, the training objective is still visible in behaviour; if they vanish, the mimicry reading loses its main evidence. One follow-up on a single reasoning model found the same pattern at higher accuracy, and a re-analysis disputes part of the maths result, so the question is open across the battery.
The debate lists this among what we still do not know: Do the failure patterns of 2023–2024 models hold for later reasoning and agentic systems?
Evidence that motivates it
- McCoy and colleagues predicted that, because language models are trained to predict the next word, their accuracy should depend on how probable the task, input and output are, even where probability should not matter; GPT-4 decoded a simple cipher with 51% accuracy when the output was a high-probability word sequence and 13% when it was low-probability. (LD01-SL-C18 v1.0)
How this study uses it: Accuracy tracks output probability in GPT-4 (51% against 13% on a cipher). - Testing OpenAI’s o1, a model trained to reason step by step, the authors of “Embers of Autoregression” found that it substantially outperformed earlier models “in many cases” but “still displays the same qualitative trends”: its accuracy at decoding a shift cipher ranged from 47% in the lowest-probability case to 92% in the highest, and it used more reasoning tokens on low-probability examples. (LD01-SL-C37 v1.0)
How this study uses it: OpenAI's o1 shows the same pattern at higher accuracy (47% to 92%). - Models fine-tuned on facts of the form “A is B” failed to answer “B is A”; GPT-4 named the parent of 79% of 1,000 celebrities, but on the 1,573 child–parent pairs this produced it named the celebrity from the parent only 33% of the time. (LD01-SL-C19 v1.0)
How this study uses it: The reversal curse in fine-tuned models and GPT-4. - Adding a clause that seems relevant but does not change the answer to grade-school maths problems cut accuracy by up to 65 percentage points across 25 models; GPT-4o scored 95.2% on the full GSM8K test set and 63.1% with the clause added, and OpenAI’s o1-preview 94.9% and 77.4%. (LD01-SL-C20 v1.0)
How this study uses it: Irrelevant clauses cut maths accuracy by up to 65 percentage points in 2024 models. - Off-the-shelf ChatGPT and GPT-4 reached only 55% and 59% accuracy on 3-digit by 3-digit multiplication, and accuracy on compositional tasks fell towards zero as complexity grew; the authors hypothesize that transformers reduce multi-step reasoning to “linearized path matching.” (LD01-SL-C21 v1.0)
How this study uses it: Accuracy on multi-digit multiplication falls towards zero as compositional depth grows. - On 11 tasks rewritten with changed premises, such as arithmetic in base 9 or 1-based indexing in code, language models performed above chance but their performance “substantially and consistently degrades” relative to the familiar versions, which the authors attribute partly to overfitting to the default tasks. (LD01-SL-C22 v1.0)
How this study uses it: Counterfactual task versions degrade performance consistently.
Each claim is shown as it read when this version was published.
A possible design
Assemble the published items of the five studies into one battery, with permission where needed, and add fresh items generated by the same procedures to limit contamination. Run it on a fixed panel of open and closed models released from 2023 to 2026 under matched prompting, with and without step-by-step reasoning and tool use. Estimate each pattern's effect size by model generation with mixed-effects models and per-item random effects, as the GSM-Symbolic re-analysis recommends, and report where effects keep their shape as accuracy rises.
Data needed
Published task items; API or weight access to models across generations; compute for evaluation; contamination checks.
Feasibility
One PhD chapter or paper. Medium; mainly API fees for closed models. Retired model versions may no longer be available, which limits the panel; record exact versions and dates. Benchmark items may have entered training data after publication, so fresh items are essential.
Has this been done?
Partly done already: related work covers some of this question (first-pass screen of 16 October 2026; not yet checked in depth).
Single follow-ups exist for one pattern or one model, but no study was found that runs these failure patterns as one battery across model generations including agentic systems.
How strong this evidence is
- Who searched and checked: no checker is recorded beyond the first-pass screen, which was part of the AI drafting (Claude (SocioLens Project Lead), AI-drafted from the LD-01 debate). Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: 1 of the 2 nearest works below was judged from an abstract, summary or search listing only; it is marked. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by one web search, arXiv record lookups and the LD-01 source scan on the date shown; replications posted as blog posts or in model cards were not searched. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- McCoy et al., o1 follow-up (LD01-SL-C37)
Full text read for this debate. Re-tests one study's tasks on one reasoning model. - Długosz, Oliveira & Díaz-Rodríguez, "The Importance of Being Statistically Earnest: A Critical Re-evaluation of GSM-Symbolic," arXiv:2605.28700 (2026) Judged from an abstract, summary or search listing only
Abstract, introduction and one footnote read; not read in full. Re-analyses the template variants on 20 open-weight models with mixed-effects models; leaves the irrelevant-clause variant out of scope.
Search queries used
embers of autoregression reasoning models o1 replication reversal curse GSM-Symbolic 2025 2026 newer modelsarXiv id_list: 2410.01792, 2605.28700LD-01 source scan for later models on limits studies (16 October 2026)
Cite this page
SocioLens project. (2026, October 16). Do the failure patterns of 2023–2024 language models persist in reasoning models? [Research opportunity RO-LD01-02, version v1]. SocioLens. https://sociolens-web.pages.dev/opportunities/ro-ld01-02/v1/
@misc{sociolens_ro_ld01_02_v1,
author = {{SocioLens project}},
title = {Do the failure patterns of 2023–2024 language models persist in reasoning models?},
year = {2026},
date = {2026-10-16},
howpublished = {\url{https://sociolens-web.pages.dev/opportunities/ro-ld01-02/v1/}},
note = {Research opportunity RO-LD01-02, version v1}
}The link points to this version, whose record never changes. For a finding, cite the original study as well: SocioLens summarizes research and does not replace it.
Provenance
RO-LD01-02 v1. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-01 debate on 2026-10-16. Not yet reviewed. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.