When four maintainers of three repositories reviewed 296 AI-written pull requests that had passed SWE-bench Verified’s automated tests, roughly half would not have been merged; on average, maintainer merge rates, scored relative to the 68% at which the same maintainers would merge the human-written patches actually merged, ran about 24 percentage points below the benchmark scores.

SocioLens project,

Claim LD06-SL-C12 (v1.0)

Research opportunities this claim motivates

What this claim leaves open for research. Each is a hypothesis about a gap, not a finding.

Do benchmark scores predict what models achieve in real deployments?Maintainers would not have merged about half of the test-passing SWE-bench Verified patches they reviewed; merge rates, scored relative to the maintainers' 68% golden-patch baseline, ran about 24 points below benchmark scores.

Provenance and status

Claim type: Estimate
Publication status: Eligible for publication, with a qualifier
Verification state: Checked against the primary source. Who checked this claim is not recorded for it; claim checks so far were AI-led, and no person has checked claims against their sources.
Does not license: A capability limit (agents could not iterate on feedback; the authors say they are not claiming benchmarks have no signal); results for other benchmarks
Limitations: A research note with less review than METR’s research posts; four maintainers; hypothetical merge decisions; 3 of SWE-bench Verified’s 12 repositories (95 of 500 issues); historical issues in their original repository state, reviewed without continuous-integration tools
Source (as cited): Whitfill, Wu, Becker & Rush, “Many SWE-bench-Passing PRs Would Not Be Merged into Main,” METR research note, 10 March 2026
Source type: Independent research report (recorded as “Research nonprofit post”)
Institutional position: Independent research nonprofit (METR)
Maturity of the evidence: Emerging evidence
Recorded date: 15 October 2026
Currentness / correction: This is the current version. No correction or concern is recorded on its sources.

Sources this version cites

Whitfill, Wu, Becker & Rush, “Many SWE-bench-Passing PRs Would Not Be Merged into Main,” METR research note, 10 March 2026 (Independent research report)
Read the original

Version

This is the current version, v1.0. Earlier versions stay available, and their records never change.

Versions: v1.0 · current

Raw record: every recorded field, with its coded values

All 25 canonical Claim fields, verbatim, with the coded values derived from them. The plain-language labels above are read from these.

Claim type (coded): ESTIMATE (as printed: “ESTIMATE”)
Publication status (coded): ELIGIBLE (QUALIFIED) (as printed: “ELIGIBLE WITH QUALIFIER”)
Verification state (coded): CONTENT_VERIFIED_PRIMARY
Source type (coded): INDEPENDENT_RESEARCH_REPORT
Maturity (coded): EMERGING

#	Canonical field	Canonical value (verbatim)	Normalized (Claim Value Vocabulary v1.0.2)
1	CLAIM ID	LD06-SL-C12	—
2	PUBLIC-FACING CLAIM	When four maintainers of three repositories reviewed 296 AI-written pull requests that had passed SWE-bench Verified’s automated tests, roughly half would not have been merged; on average, maintainer merge rates, scored relative to the 68% at which the same maintainers would merge the human-written patches actually merged, ran about 24 percentage points below the benchmark scores.	—
3	CLAIM TYPE	ESTIMATE	base_types: ESTIMATE
4	RESEARCH STRAND	A — Benchmark validity and measurement	—
5	MATURITY	EMERGING	states: EMERGING
6	SOURCE	Whitfill, Wu, Becker & Rush, “Many SWE-bench-Passing PRs Would Not Be Merged into Main,” METR research note, 10 March 2026	source_ids: SRC-SL-WHITFILL-WU-BECKER-RUSH-METR-2026-03-10
7	SOURCE TYPE	Research nonprofit post	classes: INDEPENDENT_RESEARCH_REPORT
8	INSTITUTIONAL POSITION	Independent research nonprofit (METR)	—
9	SCOPE	Three SWE-bench Verified repositories; agents from mid-2024 to mid/late 2025	—
10	DESIGN	Maintainer review of agent patches, adjusted for noise using human-written patches that were actually merged	—
11	SAMPLE / DATASET	296 AI-generated pull requests and 47 human “golden” patches	—
12	MODEL / TOOL / VERSION	Several agents released from mid-2024 to late 2025	—
13	COMPARATOR	The benchmark’s automated grader; maintainers’ decisions on golden patches	—
14	MAIN RESULT	About half of test-passing patches not mergeable; merge decisions about 24 points below SWE-bench scores, with all scores reported as a percentage of the 68% golden-patch baseline; improvement 9.6 points a year slower, a result the authors call shakier	—
15	INFERENTIAL TARGET	How far one coding benchmark’s score predicts work an organization would accept	—
16	DOES NOT LICENSE	A capability limit (agents could not iterate on feedback; the authors say they are not claiming benchmarks have no signal); results for other benchmarks	—
17	VERIFICATION STATE	CONTENT VERIFIED	state: CONTENT_VERIFIED_PRIMARY · verified_by: UNKNOWN
18	LIMITATIONS	A research note with less review than METR’s research posts; four maintainers; hypothetical merge decisions; 3 of SWE-bench Verified’s 12 repositories (95 of 500 issues); historical issues in their original repository state, reviewed without continuous-integration tools	—
19	CONTRADICTORY / SUPPORTING SOURCES	Consistent with METR’s earlier review of 15 agent pull requests (LD-03 debate) and with LD06-SL-C08	—
20	GEOGRAPHY / LANGUAGE	Not checked / English	—
21	DISTRIBUTIONAL LEDGER	Not analysed in this claim	—
22	R01 / FIELD-MAP LINKS	LD-06, D4b	—
23	WHAT WOULD CHANGE THIS CLAIM?	Larger maintainer reviews with current agents allowed to iterate	—
24	PUBLICATION STATUS	ELIGIBLE WITH QUALIFIER	state: ELIGIBLE · qualifiers: QUALIFIED
25	RECORDED DATE	2026-10-15	—

Field 22 references resolved against Field Map v0.2: LD-06 (LIVING_DEBATE); D4b (SUB_AREA)

The sources as recorded, with their record IDs and coded source types:

Pinned Sources (field 6, version-specific)

“Whitfill, Wu, Becker & Rush, “Many SWE-bench-Passing PRs Would Not Be Merged into Main,” METR research note, 10 March 2026” → Whitfill et al. (2026), “Many SWE-bench-Passing PRs Would…” (SRC-SL-WHITFILL-WU-BECKER-RUSH-METR-2026-03-10--v1.0) — Whitfill, Wu, Becker & Rush, “Many SWE-bench-Passing PRs Would Not Be Merged into Main,” METR research note, 10 March 2026; class INDEPENDENT_RESEARCH_REPORT (explicit public provenance pin)

Cited by Living Debate versions

The evaluation apparatus: Can benchmarks, leaderboards and audits carry the weight placed on them? (v0.1) LD-06

Cite this page

SocioLens project. (2026, October 15). When four maintainers of three repositories reviewed 296 AI-written pull requests that had passed SWE-bench Verified’s automated tests, roughly half would not have been merged; on average, maintainer merge rates, scored relative to the 68% at which the same maintainers would merge the human-written patches actually merged, ran about 24 percentage points below the benchmark scores [Claim LD06-SL-C12, version v1.0]. SocioLens. https://sociolens-web.pages.dev/claims/ld06-sl-c12/v1.0/

@misc{sociolens_ld06_sl_c12_v1_0,
  author = {{SocioLens project}},
  title = {When four maintainers of three repositories reviewed 296 AI-written pull requests that had passed SWE-bench Verified’s automated tests, roughly half would not have been merged; on average, maintainer merge rates, scored relative to the 68\% at which the same maintainers would merge the human-written patches actually merged, ran about 24 percentage points below the benchmark scores.},
  year = {2026},
  date = {2026-10-15},
  howpublished = {\url{https://sociolens-web.pages.dev/claims/ld06-sl-c12/v1.0/}},
  note = {Claim LD06-SL-C12, version v1.0}
}

The link points to this version, whose record never changes. For a finding, cite the original study as well: SocioLens summarizes research and does not replace it.

Research-of-record provenance

Materialized by sociolens-materializer 0.6.0 at 2026-09-28T20:00:00.000Z; normalization contract SocioLens — Claim Value Vocabulary v1.0.2 — Pre-Spike Materialization Contract (v1.0.2). Content seal SHA-256 140a110ba29dfb68aa2f247adc9d83aa9c869027f25592350579ef90b1563649.

Repository-authored inputs (not Drive exports)

SocioLens claim — LD06-SL-C12 v1.0 (v1.0) — written in the SocioLens repository by Project Lead (Claude, AI) on 2026-10-15; no Drive identity exists. Authority: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts. Block “ENTRY 12 — LD06-SL-C12” SHA-256 6beb2b381181e6480deb5d0941d8175140ceb50b05902ed68d3a72a0116d3ac1; fixture fixtures/authored/claims/LD06-SL-C12--v1.0.

Builder decisions (not Drive data)

VERSION = v1.0 — basis: Claim version taken from the repository-authored claim fixture version (SocioLens claim — LD06-SL-C12 v1.0); the first version of a new Claim of the LD-06 debate
CURRENT_POINTER = true — basis: repository-authored claim "SocioLens claim — LD06-SL-C12 v1.0" (Project Lead (Claude, AI), 2026-10-15); basis: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts
FIELD_6_SOURCE_REFERENCE = SRC-SL-WHITFILL-WU-BECKER-RUSH-METR-2026-03-10--v1.0 — basis: field 6 of ENTRY 12 — LD06-SL-C12 cites "Whitfill, Wu, Becker & Rush, “Many SWE-bench-Passing PRs Would Not Be Merged into Main,” METR research note, 10 March 2026"; the only field-6 segment of this ClaimVersion's repository-authored ledger block.
SOURCE_PROVENANCE_PIN = SRC-SL-WHITFILL-WU-BECKER-RUSH-METR-2026-03-10--v1.0 — basis: explicit version-pinned public provenance of this ClaimVersion (Amendment v1.2 §6): the pinned field-6 Source record "Whitfill, Wu, Becker & Rush, “Many SWE-bench-Passing PRs Would Not Be Merged into Main,” METR research note, 10 March 2026".