Who first detects failures in public-sector algorithmic systems, and how late?

Research question

For documented failures of public-sector algorithmic systems since 2020, who first detected the problem (internal monitoring, frontline staff, auditors, regulators, courts, journalists, advocacy groups, affected people), how long after deployment, and did detection lead to one-case correction or to system-level redesign?

Why it matters

Formal review and monitoring are offered as the reason algorithmic administration is safe. If failures are mostly found from outside, years after deployment, that reliance is misplaced and oversight design should change. A census of cancelled systems already shows that media, civil society and courts often play a role in ending them, but it does not establish who noticed first, how long problems ran, or what happens with systems that were repaired rather than cancelled.

The debate lists this among what we still do not know: How often do internal monitoring and frontline staff catch system failures before regulators, courts, journalists or advocacy groups do?

Evidence that motivates it

1. Robodebt shows that formal review institutions can exist without reliable system-level repair; the Royal Commission recommended stronger escalation, review visibility, debt-review protections and automated-decision governance. (LD05-HQ-C13 v1.0)
   Robodebt shows that formal review institutions can exist without reliable system-level repair; the claim does not say all appeals are ineffective, so the study measures how often detection leads to redesign across cases.
   Cites: Report of the Royal Commission into the Robodebt Scheme…
2. A 2020 judicial-review challenge alleging race discrimination and feedback-loop effects was followed by the Home Office’s withdrawal of its original visa Streaming Tool before a merits ruling; successor routing systems later remained in production. (LD05-HQ-C15 v1.0)
   The Home Office withdrew its original visa Streaming Tool after a judicial-review challenge and before any merits ruling, an example of an external trigger ending a system without a court finding; the study codes such triggers without treating withdrawal as proof of unlawfulness.
   Cites: ICIBI 2023 inspection; Home Office transparency/guidance…
3. Försäkringskassan withdrew an ML risk profile for parental-benefit controls during Swedish data-protection supervision in 2025; the regulator did not make a discrimination finding. (LD05-HQ-C11 v1.0)
   Försäkringskassan withdrew a risk profile during data-protection supervision without a discrimination finding, an example of a regulator-linked detection channel to be coded as distinct from a finding of harm.
   Cites: IMY, Försäkringskassan supervision decision, 2025

Claims are shown as they read on the day this pack was built; claim pages keep every earlier version.

A possible design

Build an event dataset of documented failures, suspensions, withdrawals and major redesigns of public-sector algorithmic systems from 2020 onward, starting from existing inventories (the Data Justice Lab census of cancelled systems, AlgorithmWatch reporting, court dockets, regulator decisions, audit office reports, national algorithm registers) and extending to systems that were repaired rather than cancelled. For each event, reconstruct a dated timeline from primary documents: deployment date, first documented sign of the problem, who raised it, when the agency acknowledged it, and the outcome (individual remedies only, parameter change, redesign, withdrawal). Code 'first detector' only where documents support it and mark the rest unknown. Analyse time from deployment to detection and to response by detector type, sector and country. Where possible, ask agencies by FOI whether internal monitoring had flagged the issue earlier.

Data needed

Public documents (court records, regulator decisions, audit and inspection reports, parliamentary records, news archives); FOI requests; existing inventories of cancelled systems.

Feasibility

One PhD chapter or paper. Low: desk research and FOI; costs are mainly researcher time. The design over-represents failures that became public, so it can describe how known failures surfaced but cannot estimate how many failures are caught quietly inside agencies; FOI questions on internal monitoring partly address this. Dating the 'first sign' will often be uncertain and should be reported with that uncertainty.

Has this been done?

Partly done already: related work covers some of this question (first-pass screen of 12 October 2026, checked again on 13 October 2026).

Redden et al. coded the factors behind 61 paused or cancelled systems (2020 fieldwork), including civil society critique, media investigation, legal action and internal effectiveness decisions, as non-exclusive factors. Johnson et al. (FAccT 2024) traced 40 abandoned or contested algorithms through phases starting with discovery and typed outcomes as repaired, unchanged or abandoned, but without counts by discoverer, timing or a public-sector split. Gules-Guctas analysed 71 US court dockets on benefits algorithms. A preprint on 480 general AI incidents from 2020 to 2026 finds only 5% were detected through internal monitoring, but does not split external detection by detector type, does not report public-sector results or time to detection, and does not track redesign. None establishes who detected a problem first, how long it ran before detection, or whether detection led to system-level redesign in systems that stayed in use, and no comparable public-sector dataset was found for 2020 onward.

Literature check. No study was found that codes first detector, time from deployment to detection and system-level response for public-sector algorithmic failures since 2020. The closest addition is Johnson et al. (FAccT 2024), a thematic analysis of 40 abandoned or contested algorithms that names discovery as the first of six phases and sorts outcomes into repaired, unchanged or abandoned, but it reports no counts by discoverer type, does not measure time to discovery and does not separate public-sector cases. Richards, Benn & Zilka (2025) code responses after 962 AI Incident Database incidents without detector or timing, and the 480-incident preprint already listed splits only internal versus external detection. Checked by Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert.

Where the search could not reach. Screened by web search on the date shown; general AI incident databases were not examined directly for public-sector coding (one preprint analysing such a database was found and its abstract read), the Information, Communication & Society article on litigation against automated decision-making could not be opened, and Google Scholar, SSRN and paywalled journals were not systematically searched. Absence of a hit is not proof that the work does not exist. Literature check of 2026-10-13: The last 9,000 characters of Johnson et al. (including any appendix case list) were not read, so the public-sector share of its 40 cases is unknown. Richards, Benn & Zilka was read at abstract level only. Forward citations of Gules-Guctas could not be listed with web search. Dutch and French-language searches and national algorithm-register based studies were not run; the German query found only AlgorithmWatch explainers. Absence of a hit is not proof that the work does not exist.

Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.

Nearest work found

• Redden, Brand, Sander & Warne, "Automating Public Services - Learning from Cancelled Systems," Data Justice Lab with Carnegie UK (Sept 2022)
  Identifies 61 paused or cancelled systems across Australia, Canada, Europe, New Zealand, the UK and the US and codes cancellation factors (agency effectiveness decisions 31, civil society critique 26, media investigation 24, legal action 19, among others), plus 12 case studies. Covers cancelled systems only, codes contributing factors rather than first detection or time to detection, and fieldwork ended in 2020.
• Gules-Guctas, "How Do Algorithmic Decision-Making Systems Used in Public Benefits Determinations Fail? Insights From Legal Challenges," Public Administration Review 86(2), 371–382 (2026; published online 25 Sept 2025), doi:10.1111/puar.70043 Judged from an abstract, summary or search listing only
  Analyses 71 US federal and state court dockets on disability, unemployment and nutrition benefit algorithms and argues errors are systemic. Only the abstract was read (via Crossref); court cases are one detection channel, and the study does not compare channels.
• Mumtaz, Noor & Mumtaz, "Post-Deployment Accountability in AI Governance - A Cross-Regulatory Empirical Analysis of AI Incidents," arXiv:2605.16281 (v1 Apr 2026; v3 Aug 2026) Judged from an abstract, summary or search listing only
  Checks AI Incident Database incidents from 2020 to 2026 against post-deployment provisions of the EU AI Act, the NIST framework and the GDPR, and distinguishes incidents detected through internal monitoring from externally detected ones. Only the abstract was read; it is not specific to the public sector, does not break external detection down by detector type, and reports no time to detection or redesign outcomes.
• Johnson, Moharana, Harrington, Andalibi, Heidari & Eslami, "The Fall of an Algorithm: Characterizing the Dynamics Toward Abandonment," Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (FAccT 2024), doi:10.1145/3630106.3658910; arXiv:2404.13802
  Full text read (first 100,000 characters): thematic analysis of 40 real-world cases (24 abandoned, 16 US-based) from web searches, incident databases and expert consultation; six phases (discovery, diagnosis, dissemination, dialogue, decision, death); outcomes typed as repaired, unchanged, abandoned by choice or abandoned through law or policy. Gives examples of discoverers (a Right to Know request, social media users) but no counts by discoverer type, no systematic time to discovery, no public/private split and no counts of repair versus abandonment. Closest conceptual frame for the design; a dated, coded dataset is what remains.
• Richards, Benn & Zilka, "From Incidents to Insights: Patterns of Responsibility following AI Harms," arXiv:2505.04291 (7 May 2025) Judged from an abstract, summary or search listing only
  Abstract read: mixed-methods analysis of 962 AI Incident Database incidents and 4,743 reports on how developers, deployers, victims and lawmakers respond after AI harms. All sectors; the abstract mentions no detector coding, time to detection or public-sector breakdown.
• Kaun, "Temporalities of welfare automation: On timing, belatedness, and perpetual emergence," Time & Society (2025), doi:10.1177/0961463X251313572 Judged from an abstract, summary or search listing only
  Seen in search results only (not opened). Title indicates a conceptual analysis of timing and belatedness in welfare automation, relevant to the time-to-detection question; not known to code a dataset.

References

Sources behind the motivating claims:

• Report of the Royal Commission into the Robodebt Scheme, 2023 (SRC-LD05-ROBODEBT-ROYAL-COMMISSION-2023, one of the original documents; the reference cites several documents and this link opens only one)
• ICIBI 2023 inspection; Home Office transparency/guidance; JCWI/Foxglove litigation record for claimant position (SRC-R04-HOME-OFFICE-ROUTING-SOURCE-SET, no checked link to the original yet)
• IMY, Försäkringskassan supervision decision, 2025 (SRC-LD05-IMY-2025-11828, original)

BibTeX

The first entry cites this opportunity; then its nearest work; then the sources above. Nearest work and sources are recorded as reference strings, so each goes in whole as the title; add authors and year from the original before you cite it.

@misc{sociolens_ro_ld05_06_v2,
  author = {{SocioLens project}},
  title = {Who first detects failures in public-sector algorithmic systems, and how late?},
  year = {2026},
  date = {2026-10-13},
  howpublished = {\url{https://sociolens-web.pages.dev/opportunities/ro-ld05-06/v2/}},
  note = {Research opportunity RO-LD05-06, version v2}
}

@misc{ro_ld05_06_nearest_1,
  title = {Redden, Brand, Sander \& Warne, "Automating Public Services - Learning from Cancelled Systems," Data Justice Lab with Carnegie UK (Sept 2022)},
  url = {https://carnegieuk.org/wp-content/uploads/2024/10/Automating-Public-Services-Learning-from-Cancelled-Systems-Final-Full-Report-2.pdf},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld05_06_nearest_2,
  title = {Gules-Guctas, "How Do Algorithmic Decision-Making Systems Used in Public Benefits Determinations Fail? Insights From Legal Challenges," Public Administration Review 86(2), 371–382 (2026; published online 25 Sept 2025), doi:10.1111/puar.70043},
  url = {https://doi.org/10.1111/puar.70043},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{ro_ld05_06_nearest_3,
  title = {Mumtaz, Noor \& Mumtaz, "Post-Deployment Accountability in AI Governance - A Cross-Regulatory Empirical Analysis of AI Incidents," arXiv:2605.16281 (v1 Apr 2026; v3 Aug 2026)},
  url = {https://arxiv.org/abs/2605.16281},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{ro_ld05_06_nearest_4,
  title = {Johnson, Moharana, Harrington, Andalibi, Heidari \& Eslami, "The Fall of an Algorithm: Characterizing the Dynamics Toward Abandonment," Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (FAccT 2024), doi:10.1145/3630106.3658910; arXiv:2404.13802},
  url = {https://doi.org/10.1145/3630106.3658910},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld05_06_nearest_5,
  title = {Richards, Benn \& Zilka, "From Incidents to Insights: Patterns of Responsibility following AI Harms," arXiv:2505.04291 (7 May 2025)},
  url = {https://arxiv.org/abs/2505.04291},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{ro_ld05_06_nearest_6,
  title = {Kaun, "Temporalities of welfare automation: On timing, belatedness, and perpetual emergence," Time \& Society (2025), doi:10.1177/0961463X251313572},
  url = {https://doi.org/10.1177/0961463X251313572},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{src_ld05_robodebt_royal_commission_2023,
  title = {Report of the Royal Commission into the Robodebt Scheme, 2023},
  url = {https://robodebt.royalcommission.gov.au/node/481},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; the link opens only one of the several documents this reference cites}
}

@misc{src_r04_home_office_routing_source_set,
  title = {ICIBI 2023 inspection; Home Office transparency/guidance; JCWI/Foxglove litigation record for claimant position},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_ld05_imy_2025_11828,
  title = {IMY, Försäkringskassan supervision decision, 2025},
  url = {https://www.imy.se/globalassets/dokument/beslut/2025/beslut-tillsyn-forsakringskassan.pdf},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}