In a 2025 randomized trial with 16 experienced open-source developers working on 246 tasks in their own projects, allowing AI tools increased completion time by 19%, although the developers estimated it had cut their time by 20%; METR later called its 2026 follow-up data an unreliable signal because of selection.

SocioLens project,

Claim LD04-SL-C07 (v1.0)

Provenance and status

Claim type: Estimate
Publication status: Eligible for publication, with qualifiers: preprint
Verification state: Checked against the primary source; the source is a preprint. Who checked this claim is not recorded for it; claim checks so far were AI-led, and no person has checked claims against their sources.
Does not license: A general claim that AI slows experts; any estimate for current tools
Limitations: Very small sample; one kind of setting; tools have changed; METR believes developers are now more sped up, and its own 2026 redesign post treats later estimates as unreliable
Source (as cited): Becker, Rush, Barnes & Rein, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity,” arXiv:2507.09089, July 2025; METR, “We are Changing our Developer Productivity Experiment Design,” 24 February 2026
Source type: Preprint, Independent research report (recorded as “Preprint + research nonprofit post”)
Institutional position: Independent research nonprofit
Maturity of the evidence: Contested evidence
Recorded date: 14 October 2026
Currentness / correction: This is the current version. No correction or concern is recorded on its sources.

Sources this version cites

Version

This page shows version v1.0. Its record never changes, even when newer evidence arrives; the site may add a note to the page or withhold a figure not cleared for public use.

This pinned version is also the current version.

Versions: v1.0 · current

Raw record: every recorded field, with its coded values

All 25 canonical Claim fields, verbatim, with the coded values derived from them. The plain-language labels above are read from these.

Claim type (coded): ESTIMATE (as printed: “ESTIMATE”)
Publication status (coded): ELIGIBLE (PREPRINT) (as printed: “ELIGIBLE WITH PREPRINT QUALIFIER”)
Verification state (coded): CONTENT_VERIFIED_PRIMARY (PREPRINT)
Source type (coded): PREPRINT, INDEPENDENT_RESEARCH_REPORT
Maturity (coded): CONTESTED

#	Canonical field	Canonical value (verbatim)	Normalized (Claim Value Vocabulary v1.0.2)
1	CLAIM ID	LD04-SL-C07	—
2	PUBLIC-FACING CLAIM	In a 2025 randomized trial with 16 experienced open-source developers working on 246 tasks in their own projects, allowing AI tools increased completion time by 19%, although the developers estimated it had cut their time by 20%; METR later called its 2026 follow-up data an unreliable signal because of selection.	—
3	CLAIM TYPE	ESTIMATE	base_types: ESTIMATE
4	RESEARCH STRAND	A — Performance while the tool is in use	—
5	MATURITY	CONTESTED	states: CONTESTED
6	SOURCE	Becker, Rush, Barnes & Rein, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity,” arXiv:2507.09089, July 2025; METR, “We are Changing our Developer Productivity Experiment Design,” 24 February 2026	source_ids: SRC-SL-BECKER-RUSH-BARNES-REIN-ARXIV-2507-09089, SRC-SL-METR-DEVELOPER-STUDY-DESIGN-2026-02-24
7	SOURCE TYPE	Preprint + research nonprofit post	classes: PREPRINT, INDEPENDENT_RESEARCH_REPORT
8	INSTITUTIONAL POSITION	Independent research nonprofit	—
9	SCOPE	Mature open-source repositories; developers with about five years on their projects; early-2025 tools	—
10	DESIGN	Randomized controlled trial at task level	—
11	SAMPLE / DATASET	16 developers, 246 tasks	—
12	MODEL / TOOL / VERSION	Early-2025 AI coding tools	—
13	COMPARATOR	Tasks on which AI use was not allowed	—
14	MAIN RESULT	Completion time +19% (CI +2% to +39%) with AI allowed; developers estimated a 20% reduction; in 2026 METR described its new data as only very weak evidence because developers unwilling to work without AI opted out and others withheld tasks	—
15	INFERENTIAL TARGET	Assisted speed of highly experienced developers on familiar codebases	—
16	DOES NOT LICENSE	A general claim that AI slows experts; any estimate for current tools	—
17	VERIFICATION STATE	CONTENT VERIFIED PREPRINT	state: CONTENT_VERIFIED_PRIMARY · qualifiers: PREPRINT · verified_by: UNKNOWN
18	LIMITATIONS	Very small sample; one kind of setting; tools have changed; METR believes developers are now more sped up, and its own 2026 redesign post treats later estimates as unreliable	—
19	CONTRADICTORY / SUPPORTING SOURCES	Contrasts with large speed gains in lab coding tasks; consistent with meta-analytic evidence that controlled-experiment gains shrink in real settings	—
20	GEOGRAPHY / LANGUAGE	Not checked / English	—
21	DISTRIBUTIONAL LEDGER	The most experienced may gain least, or lose, on their own familiar work	—
22	R01 / FIELD-MAP LINKS	LD-04, D2	—
23	WHAT WOULD CHANGE THIS CLAIM?	A larger, well-selected trial with current tools	—
24	PUBLICATION STATUS	ELIGIBLE WITH PREPRINT QUALIFIER	state: ELIGIBLE · qualifiers: PREPRINT
25	RECORDED DATE	2026-10-14	—

Field 22 references resolved against Field Map v0.2: LD-04 (LIVING_DEBATE); D2 (DOMAIN)

The sources as recorded, with their record IDs and coded source types:

Pinned Sources (field 6, version-specific)

“Becker, Rush, Barnes & Rein, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity,” arXiv:2507.09089, July 2025” → Becker et al. (2025), “Measuring the Impact of Early-2025…” (SRC-SL-BECKER-RUSH-BARNES-REIN-ARXIV-2507-09089--v1.0) — Becker, Rush, Barnes & Rein, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity,” arXiv:2507.09089, July 2025; class PREPRINT (explicit public provenance pin)
“METR, “We are Changing our Developer Productivity Experiment Design,” 24 February 2026” → METR (2026), “We are Changing our Developer Productivity…” (SRC-SL-METR-DEVELOPER-STUDY-DESIGN-2026-02-24--v1.0) — METR, “We are Changing our Developer Productivity Experiment Design,” 24 February 2026; class INDEPENDENT_RESEARCH_REPORT (explicit public provenance pin)

Cited by Living Debate versions

Cite this page

SocioLens project. (2026, October 14). In a 2025 randomized trial with 16 experienced open-source developers working on 246 tasks in their own projects, allowing AI tools increased completion time by 19%, although the developers estimated it had cut their time by 20%; METR later called its 2026 follow-up data an unreliable signal because of selection [Claim LD04-SL-C07, version v1.0]. SocioLens. https://sociolens-web.pages.dev/claims/ld04-sl-c07/v1.0/

@misc{sociolens_ld04_sl_c07_v1_0,
  author = {{SocioLens project}},
  title = {In a 2025 randomized trial with 16 experienced open-source developers working on 246 tasks in their own projects, allowing AI tools increased completion time by 19\%, although the developers estimated it had cut their time by 20\%; METR later called its 2026 follow-up data an unreliable signal because of selection.},
  year = {2026},
  date = {2026-10-14},
  howpublished = {\url{https://sociolens-web.pages.dev/claims/ld04-sl-c07/v1.0/}},
  note = {Claim LD04-SL-C07, version v1.0}
}

The link points to this version, whose record never changes. For a finding, cite the original study as well: SocioLens summarizes research and does not replace it.

Research-of-record provenance

Materialized by sociolens-materializer 0.6.0 at 2026-09-28T20:00:00.000Z; normalization contract SocioLens — Claim Value Vocabulary v1.0.2 — Pre-Spike Materialization Contract (v1.0.2). Content seal SHA-256 7abe93a263e9b6337e9e9e3bbcaac2c415822013d9dc45730a68188c5778d643.

Repository-authored inputs (not Drive exports)

SocioLens claim — LD04-SL-C07 v1.0 (v1.0) — written in the SocioLens repository by Project Lead (Claude, AI) on 2026-10-14; no Drive identity exists. Authority: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 14 October 2026 to build the LD-04 debate; a new Claim, first version, written from the project's LD-04 source notes, checked against full texts or official abstracts. Block “ENTRY 07 — LD04-SL-C07” SHA-256 34bf84fff7760b7961fac666a0aaa1c78f2b796fe3190dd2b257ca057e636091; fixture fixtures/authored/claims/LD04-SL-C07--v1.0.

Builder decisions (not Drive data)

VERSION = v1.0 — basis: Claim version taken from the repository-authored claim fixture version (SocioLens claim — LD04-SL-C07 v1.0); the first version of a new Claim of the LD-04 debate
CURRENT_POINTER = true — basis: repository-authored claim "SocioLens claim — LD04-SL-C07 v1.0" (Project Lead (Claude, AI), 2026-10-14); basis: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 14 October 2026 to build the LD-04 debate; a new Claim, first version, written from the project's LD-04 source notes, checked against full texts or official abstracts
FIELD_6_SOURCE_REFERENCE = SRC-SL-BECKER-RUSH-BARNES-REIN-ARXIV-2507-09089--v1.0 — basis: field 6 of ENTRY 07 — LD04-SL-C07 cites "Becker, Rush, Barnes & Rein, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity,” arXiv:2507.09089, July 2025"; the first field-6 segment of this ClaimVersion's repository-authored ledger block.
FIELD_6_SOURCE_REFERENCE = SRC-SL-METR-DEVELOPER-STUDY-DESIGN-2026-02-24--v1.0 — basis: field 6 of ENTRY 07 — LD04-SL-C07 cites "METR, “We are Changing our Developer Productivity Experiment Design,” 24 February 2026"; the second field-6 segment of this ClaimVersion's repository-authored ledger block.
SOURCE_PROVENANCE_PIN = SRC-SL-BECKER-RUSH-BARNES-REIN-ARXIV-2507-09089--v1.0 — basis: explicit version-pinned public provenance of this ClaimVersion (Amendment v1.2 §6): the pinned field-6 Source record "Becker, Rush, Barnes & Rein, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity,” arXiv:2507.09089, July 2025".
SOURCE_PROVENANCE_PIN = SRC-SL-METR-DEVELOPER-STUDY-DESIGN-2026-02-24--v1.0 — basis: explicit version-pinned public provenance of this ClaimVersion (Amendment v1.2 §6): the pinned field-6 Source record "METR, “We are Changing our Developer Productivity Experiment Design,” 24 February 2026".