Skip to content
Hot eventDeveloping

Harvard Study Finds Claude Fable 5 Ban Barely Affected Developer Merge Rates

1 reports1 reporting sourcesUpdated 1 h ago

Start here

AI digest

Harvard researchers found that losing access to Claude Fable 5 during a 19-day ban barely changed how often developers got their code merged. Tracking 297 Fable 5 users before, during and after the ban and comparing them with 8,908 developers who kept using the older Opus 4.8, the study measured the share of submitted code changes accepted into GitHub projects within seven days; access to the model raised that share by just 1.4 percentage points, a difference the paper calls not statistically significant. Across five Claude upgrades in 2026, covering 1,208 developers and more than 15,000 code submissions, the researchers found no significant gain in merge rates or weekly completed changes, even where benchmark scores rose 5 to 18 points. The authors suggest developers' tasks already fall within what older models can do, so benchmark scores may not reflect real work productivity. The paper acknowledges GitHub merges are an imperfect productivity measure but calls the ban a rare chance to test model switching under real working conditions.

Generated from reports · updated 1 h ago

Report timeline

Follow the reports to see the event from different sides.

November 18
  1. Hacker News · AI
    Harvard study finds Claude Fable ban barely changed developer merge rates

    A Harvard study tracking 297 Claude Fable 5 users through the 19-day ban found that access to the model raised the share of pull requests merged within seven days by just 1.4 percentage points, a difference the paper calls not statistically significant. Across five Claude upgrades in 2026 covering 1,208 developers and more than 15,000 code submissions, no significant gain appeared in merge rates or weekly completed changes, even where benchmark scores rose 5 to 18 points. The authors suggest the tasks developers hand to agents already fall within what older models can do, so benchmark scores may not reflect real work productivity.

Heat for this event

Not enough continuous observations yet to draw a trend.