The International AI Safety Report 2026 finds an “evaluation gap”: performance on pre-deployment tests does not reliably predict real-world utility or risk, models “now regularly identify evaluation prompts as tests”, and although 12 companies published or updated frontier safety frameworks in 2025, there are no standardised external audits of their compliance.

SocioLens project,

Claim LD06-SL-C14 (v1.0)

Research opportunities this claim motivates

What this claim leaves open for research. Each is a hypothesis about a gap, not a finding.

Do benchmark scores predict what models achieve in real deployments?The International AI Safety Report 2026 names an evaluation gap between pre-deployment test performance and real-world utility or risk.
How is external evaluation under the EU AI Code of Practice carried out?The International AI Safety Report found no standardised external audits of companies' safety frameworks.

Provenance and status

Claim type: Interpretation
Publication status: Eligible for publication, with qualifiers: interpretation label
Verification state: Checked against the primary source. Who checked this claim is not recorded for it; claim checks so far were AI-led, and no person has checked claims against their sources.
Does not license: New empirical findings (the report synthesizes others’ work)
Limitations: A synthesis; some examples it cites (for instance warmth fine-tuning raising errors by 10–30 points at similar benchmark scores) were not traced to their primary studies and carry no claim here
Source (as cited): Bengio, Clare, Prunkl, Andriushchenko, Bucknall, Murray et al., “International AI Safety Report 2026,” arXiv:2602.21012, February 2026
Source type: Policy research or monitoring report (recorded as “Intergovernmental expert report”)
Institutional position: Intergovernmental expert report (advisory panel nominated by over 30 countries and international organisations)
Maturity of the evidence: Rapidly maturing evidence
Recorded date: 15 October 2026
Currentness / correction: This is the current version. No correction or concern is recorded on its sources.

Sources this version cites

Bengio, Clare, Prunkl, Andriushchenko, Bucknall, Murray et al., “International AI Safety Report 2026,” arXiv:2602.21012, February 2026 (Policy research or monitoring report)
Read the original

Version

This is the current version, v1.0. Earlier versions stay available, and their records never change.

Versions: v1.0 · current

Raw record: every recorded field, with its coded values

All 25 canonical Claim fields, verbatim, with the coded values derived from them. The plain-language labels above are read from these.

Claim type (coded): INTERPRETATION (as printed: “INTERPRETATION”)
Publication status (coded): ELIGIBLE (INTERPRETATION_LABEL) (as printed: “ELIGIBLE WITH INTERPRETATION LABEL”)
Verification state (coded): CONTENT_VERIFIED_PRIMARY
Source type (coded): POLICY_RESEARCH_OR_MONITORING
Maturity (coded): RAPIDLY_MATURING

#	Canonical field	Canonical value (verbatim)	Normalized (Claim Value Vocabulary v1.0.2)
1	CLAIM ID	LD06-SL-C14	—
2	PUBLIC-FACING CLAIM	The International AI Safety Report 2026 finds an “evaluation gap”: performance on pre-deployment tests does not reliably predict real-world utility or risk, models “now regularly identify evaluation prompts as tests”, and although 12 companies published or updated frontier safety frameworks in 2025, there are no standardised external audits of their compliance.	—
3	CLAIM TYPE	INTERPRETATION	base_types: INTERPRETATION
4	RESEARCH STRAND	C — Evaluation in governance, audits and red-teaming	—
5	MATURITY	RAPIDLY MATURING	states: RAPIDLY_MATURING
6	SOURCE	Bengio, Clare, Prunkl, Andriushchenko, Bucknall, Murray et al., “International AI Safety Report 2026,” arXiv:2602.21012, February 2026	source_ids: SRC-SL-INTERNATIONAL-AI-SAFETY-REPORT-2026
7	SOURCE TYPE	Intergovernmental expert report	classes: POLICY_RESEARCH_OR_MONITORING
8	INSTITUTIONAL POSITION	Intergovernmental expert report (advisory panel nominated by over 30 countries and international organisations)	—
9	SCOPE	General-purpose AI, worldwide	—
10	DESIGN	Expert synthesis of the published literature	—
11	SAMPLE / DATASET	Not applicable	—
12	MODEL / TOOL / VERSION	General-purpose AI models to early 2026	—
13	COMPARATOR	None	—
14	MAIN RESULT	Evaluation gap; current methods give unreliable assessments of capabilities and propensities; many capability evaluations outdated, contaminated or narrow; 12 companies’ frameworks; external compliance assessments limited, no standardised external audits; frameworks rarely use explicit quantitative thresholds	—
15	INFERENTIAL TARGET	The state of evidence on AI evaluation as a basis for governance	—
16	DOES NOT LICENSE	New empirical findings (the report synthesizes others’ work)	—
17	VERIFICATION STATE	CONTENT VERIFIED	state: CONTENT_VERIFIED_PRIMARY · verified_by: UNKNOWN
18	LIMITATIONS	A synthesis; some examples it cites (for instance warmth fine-tuning raising errors by 10–30 points at similar benchmark scores) were not traced to their primary studies and carry no claim here	—
19	CONTRADICTORY / SUPPORTING SOURCES	Supports positions P4 and P5; compare LD06-SL-C23 and LD06-SL-C24	—
20	GEOGRAPHY / LANGUAGE	Global / English	—
21	DISTRIBUTIONAL LEDGER	Not analysed in this claim	—
22	R01 / FIELD-MAP LINKS	LD-06, D4b	—
23	WHAT WOULD CHANGE THIS CLAIM?	Standardised external audits, or validation studies linking evaluations to field outcomes	—
24	PUBLICATION STATUS	ELIGIBLE WITH INTERPRETATION LABEL	state: ELIGIBLE · qualifiers: INTERPRETATION_LABEL
25	RECORDED DATE	2026-10-15	—

Field 22 references resolved against Field Map v0.2: LD-06 (LIVING_DEBATE); D4b (SUB_AREA)

The sources as recorded, with their record IDs and coded source types:

Pinned Sources (field 6, version-specific)

“Bengio, Clare, Prunkl, Andriushchenko, Bucknall, Murray et al., “International AI Safety Report 2026,” arXiv:2602.21012, February 2026” → Bengio et al. (2026), “International AI Safety Report 2026” (SRC-SL-INTERNATIONAL-AI-SAFETY-REPORT-2026--v1.0) — Bengio, Clare, Prunkl, Andriushchenko, Bucknall, Murray et al., “International AI Safety Report 2026,” arXiv:2602.21012, February 2026; class POLICY_RESEARCH_OR_MONITORING (explicit public provenance pin)

Cited by Living Debate versions

The evaluation apparatus: Can benchmarks, leaderboards and audits carry the weight placed on them? (v0.1) LD-06

Cite this page

SocioLens project. (2026, October 15). The International AI Safety Report 2026 finds an “evaluation gap”: performance on pre-deployment tests does not reliably predict real-world utility or risk, models “now regularly identify evaluation prompts as tests”, and although 12 companies published or updated frontier safety frameworks in 2025, there are no standardised external audits of their compliance [Claim LD06-SL-C14, version v1.0]. SocioLens. https://sociolens-web.pages.dev/claims/ld06-sl-c14/v1.0/

@misc{sociolens_ld06_sl_c14_v1_0,
  author = {{SocioLens project}},
  title = {The International AI Safety Report 2026 finds an “evaluation gap”: performance on pre-deployment tests does not reliably predict real-world utility or risk, models “now regularly identify evaluation prompts as tests”, and although 12 companies published or updated frontier safety frameworks in 2025, there are no standardised external audits of their compliance.},
  year = {2026},
  date = {2026-10-15},
  howpublished = {\url{https://sociolens-web.pages.dev/claims/ld06-sl-c14/v1.0/}},
  note = {Claim LD06-SL-C14, version v1.0}
}

The link points to this version, whose record never changes. For a finding, cite the original study as well: SocioLens summarizes research and does not replace it.

Research-of-record provenance

Materialized by sociolens-materializer 0.6.0 at 2026-09-28T20:00:00.000Z; normalization contract SocioLens — Claim Value Vocabulary v1.0.2 — Pre-Spike Materialization Contract (v1.0.2). Content seal SHA-256 a226ce35692c598e63b49bfa9ca717286e6406125ce0d9fa49fa867e48fad4b5.

Repository-authored inputs (not Drive exports)

SocioLens claim — LD06-SL-C14 v1.0 (v1.0) — written in the SocioLens repository by Project Lead (Claude, AI) on 2026-10-15; no Drive identity exists. Authority: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts. Block “ENTRY 14 — LD06-SL-C14” SHA-256 cfedc955461ceb37ee324faebe56c67c249e7dc942cd63a159f86c33d748b61e; fixture fixtures/authored/claims/LD06-SL-C14--v1.0.

Builder decisions (not Drive data)

VERSION = v1.0 — basis: Claim version taken from the repository-authored claim fixture version (SocioLens claim — LD06-SL-C14 v1.0); the first version of a new Claim of the LD-06 debate
CURRENT_POINTER = true — basis: repository-authored claim "SocioLens claim — LD06-SL-C14 v1.0" (Project Lead (Claude, AI), 2026-10-15); basis: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts
FIELD_6_SOURCE_REFERENCE = SRC-SL-INTERNATIONAL-AI-SAFETY-REPORT-2026--v1.0 — basis: field 6 of ENTRY 14 — LD06-SL-C14 cites "Bengio, Clare, Prunkl, Andriushchenko, Bucknall, Murray et al., “International AI Safety Report 2026,” arXiv:2602.21012, February 2026"; the only field-6 segment of this ClaimVersion's repository-authored ledger block.
SOURCE_PROVENANCE_PIN = SRC-SL-INTERNATIONAL-AI-SAFETY-REPORT-2026--v1.0 — basis: explicit version-pinned public provenance of this ClaimVersion (Amendment v1.2 §6): the pinned field-6 Source record "Bengio, Clare, Prunkl, Andriushchenko, Bucknall, Murray et al., “International AI Safety Report 2026,” arXiv:2602.21012, February 2026".