An analysis of MMLU found that 28% of its questions require culturally sensitive knowledge and that 84.9% of its geography questions concern North America or Europe; model rankings change depending on whether the culturally sensitive questions are included, so machine-translated versions distort multilingual comparisons.

SocioLens project,

Claim LD06-SL-C11 (v1.0)

Research opportunities this claim motivates

What this claim leaves open for research. Each is a hypothesis about a gap, not a finding.

Who builds and checks China's compliance test banks for generative AI?Benchmark content reflects where it was made, which bears on comparing Chinese and Western safety testing.

Provenance and status

Claim type: Estimate
Publication status: Eligible for publication, with qualifiers: institutional position
Verification state: Checked against the abstract of the primary source only. Who checked this claim is not recorded for it; claim checks so far were AI-led, and no person has checked claims against their sources.
Does not license: A judgement on multilingual benchmarks other than MMLU
Limitations: Read in abstract only; authors’ lab develops models; the full text, checked for this point only, estimates the 28% from a 2,850-question annotated sample
Source (as cited): Singh, Romanou, Fourrier, Adelani, Ngui, Vila-Suero et al., “Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation,” arXiv:2412.03304 (v2), 2025
Source type: Preprint (recorded as “Preprint”)
Institutional position: AI developer-affiliated research lab (Cohere For AI) with academic co-authors
Maturity of the evidence: Emerging evidence
Recorded date: 15 October 2026
Currentness / correction: This is the current version. No correction or concern is recorded on its sources.

Sources this version cites

Singh, Romanou, Fourrier, Adelani, Ngui, Vila-Suero et al., “Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation,” arXiv:2412.03304 (v2), 2025 (Preprint)
Read the original

Version

This is the current version, v1.0. Earlier versions stay available, and their records never change.

Versions: v1.0 · current

Raw record: every recorded field, with its coded values

All 25 canonical Claim fields, verbatim, with the coded values derived from them. The plain-language labels above are read from these.

Claim type (coded): ESTIMATE (as printed: “ESTIMATE”)
Publication status (coded): ELIGIBLE (INSTITUTIONAL_POSITION) (as printed: “ELIGIBLE WITH INSTITUTIONAL-POSITION QUALIFIER”)
Verification state (coded): CONTENT_VERIFIED_PRIMARY (ABSTRACT_ONLY)
Source type (coded): PREPRINT
Maturity (coded): EMERGING

#	Canonical field	Canonical value (verbatim)	Normalized (Claim Value Vocabulary v1.0.2)
1	CLAIM ID	LD06-SL-C11	—
2	PUBLIC-FACING CLAIM	An analysis of MMLU found that 28% of its questions require culturally sensitive knowledge and that 84.9% of its geography questions concern North America or Europe; model rankings change depending on whether the culturally sensitive questions are included, so machine-translated versions distort multilingual comparisons.	—
3	CLAIM TYPE	ESTIMATE	base_types: ESTIMATE
4	RESEARCH STRAND	A — Benchmark validity and measurement	—
5	MATURITY	EMERGING	states: EMERGING
6	SOURCE	Singh, Romanou, Fourrier, Adelani, Ngui, Vila-Suero et al., “Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation,” arXiv:2412.03304 (v2), 2025	source_ids: SRC-SL-SINGH-ET-AL-GLOBAL-MMLU-ARXIV-2412-03304
7	SOURCE TYPE	Preprint	classes: PREPRINT
8	INSTITUTIONAL POSITION	AI developer-affiliated research lab (Cohere For AI) with academic co-authors	—
9	SCOPE	MMLU and translations into 42 languages	—
10	DESIGN	Annotation of cultural and regional knowledge; evaluation of models on full and subset versions	—
11	SAMPLE / DATASET	MMLU questions, annotated by professional and community annotators	—
12	MODEL / TOOL / VERSION	State-of-the-art open and proprietary models as of late 2024	—
13	COMPARATOR	Culturally sensitive versus culturally agnostic subsets	—
14	MAIN RESULT	28% of questions culturally sensitive; 84.9% of geographic questions North American or European; rankings shift between subsets	—
15	INFERENTIAL TARGET	Whether an English-made benchmark measures the same thing across languages and regions	—
16	DOES NOT LICENSE	A judgement on multilingual benchmarks other than MMLU	—
17	VERIFICATION STATE	CONTENT VERIFIED ABSTRACT	state: CONTENT_VERIFIED_PRIMARY · qualifiers: ABSTRACT_ONLY · verified_by: UNKNOWN
18	LIMITATIONS	Read in abstract only; authors’ lab develops models; the full text, checked for this point only, estimates the 28% from a 2,850-question annotated sample	—
19	CONTRADICTORY / SUPPORTING SOURCES	Compare the cultural biases found in ThaiExam: LD06-SL-C01	—
20	GEOGRAPHY / LANGUAGE	Global / multilingual	—
21	DISTRIBUTIONAL LEDGER	Benchmark content favours Western-centric knowledge	—
22	R01 / FIELD-MAP LINKS	LD-06, D4b	—
23	WHAT WOULD CHANGE THIS CLAIM?	Evidence that rankings are stable across culturally balanced subsets	—
24	PUBLICATION STATUS	ELIGIBLE WITH INSTITUTIONAL-POSITION QUALIFIER	state: ELIGIBLE · qualifiers: INSTITUTIONAL_POSITION
25	RECORDED DATE	2026-10-15	—

Field 22 references resolved against Field Map v0.2: LD-06 (LIVING_DEBATE); D4b (SUB_AREA)

The sources as recorded, with their record IDs and coded source types:

Pinned Sources (field 6, version-specific)

“Singh, Romanou, Fourrier, Adelani, Ngui, Vila-Suero et al., “Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation,” arXiv:2412.03304 (v2), 2025” → Singh et al. (2025), “Global MMLU: Understanding and…” (SRC-SL-SINGH-ET-AL-GLOBAL-MMLU-ARXIV-2412-03304--v1.0) — Singh, Romanou, Fourrier, Adelani, Ngui, Vila-Suero et al., “Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation,” arXiv:2412.03304 (v2), 2025; class PREPRINT (explicit public provenance pin)

Cited by Living Debate versions

The evaluation apparatus: Can benchmarks, leaderboards and audits carry the weight placed on them? (v0.1) LD-06

Cite this page

SocioLens project. (2026, October 15). An analysis of MMLU found that 28% of its questions require culturally sensitive knowledge and that 84.9% of its geography questions concern North America or Europe; model rankings change depending on whether the culturally sensitive questions are included, so machine-translated versions distort multilingual comparisons [Claim LD06-SL-C11, version v1.0]. SocioLens. https://sociolens-web.pages.dev/claims/ld06-sl-c11/v1.0/

@misc{sociolens_ld06_sl_c11_v1_0,
  author = {{SocioLens project}},
  title = {An analysis of MMLU found that 28\% of its questions require culturally sensitive knowledge and that 84.9\% of its geography questions concern North America or Europe; model rankings change depending on whether the culturally sensitive questions are included, so machine-translated versions distort multilingual comparisons.},
  year = {2026},
  date = {2026-10-15},
  howpublished = {\url{https://sociolens-web.pages.dev/claims/ld06-sl-c11/v1.0/}},
  note = {Claim LD06-SL-C11, version v1.0}
}

The link points to this version, whose record never changes. For a finding, cite the original study as well: SocioLens summarizes research and does not replace it.

Research-of-record provenance

Materialized by sociolens-materializer 0.6.0 at 2026-09-28T20:00:00.000Z; normalization contract SocioLens — Claim Value Vocabulary v1.0.2 — Pre-Spike Materialization Contract (v1.0.2). Content seal SHA-256 8df269b8f82b90254ab17156b224cc2748f7fd1235eb1d6ff15960cb81066fa2.

Repository-authored inputs (not Drive exports)

SocioLens claim — LD06-SL-C11 v1.0 (v1.0) — written in the SocioLens repository by Project Lead (Claude, AI) on 2026-10-15; no Drive identity exists. Authority: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts. Block “ENTRY 11 — LD06-SL-C11” SHA-256 35aa7646005730608728aeabc230d89c2f2784692e6cdbab60665a4ed980ee75; fixture fixtures/authored/claims/LD06-SL-C11--v1.0.

Builder decisions (not Drive data)

VERSION = v1.0 — basis: Claim version taken from the repository-authored claim fixture version (SocioLens claim — LD06-SL-C11 v1.0); the first version of a new Claim of the LD-06 debate
CURRENT_POINTER = true — basis: repository-authored claim "SocioLens claim — LD06-SL-C11 v1.0" (Project Lead (Claude, AI), 2026-10-15); basis: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts
FIELD_6_SOURCE_REFERENCE = SRC-SL-SINGH-ET-AL-GLOBAL-MMLU-ARXIV-2412-03304--v1.0 — basis: field 6 of ENTRY 11 — LD06-SL-C11 cites "Singh, Romanou, Fourrier, Adelani, Ngui, Vila-Suero et al., “Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation,” arXiv:2412.03304 (v2), 2025"; the only field-6 segment of this ClaimVersion's repository-authored ledger block.
SOURCE_PROVENANCE_PIN = SRC-SL-SINGH-ET-AL-GLOBAL-MMLU-ARXIV-2412-03304--v1.0 — basis: explicit version-pinned public provenance of this ClaimVersion (Amendment v1.2 §6): the pinned field-6 Source record "Singh, Romanou, Fourrier, Adelani, Ngui, Vila-Suero et al., “Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation,” arXiv:2412.03304 (v2), 2025".