Harvard Study Finds Claude Fable 5 Ban Barely Affected Developer Merge Rates
Start here
Harvard researchers found that losing access to Claude Fable 5 during a 19-day ban barely changed how often developers got their code merged. Tracking 297 Fable 5 users before, during and after the ban and comparing them with 8,908 developers who kept using the older Opus 4.8, the study measured the share of submitted code changes accepted into GitHub projects within seven days; access to the model raised that share by just 1.4 percentage points, a difference the paper calls not statistically significant. Across five Claude upgrades in 2026, covering 1,208 developers and more than 15,000 code submissions, the researchers found no significant gain in merge rates or weekly completed changes, even where benchmark scores rose 5 to 18 points. The authors suggest developers' tasks already fall within what older models can do, so benchmark scores may not reflect real work productivity. The paper acknowledges GitHub merges are an imperfect productivity measure but calls the ban a rare chance to test model switching under real working conditions.
Generated from reports · updated 1 h ago
Report timeline
Follow the reports to see the event from different sides.
- Hacker News · AIHarvard study finds Claude Fable ban barely changed developer merge rates
A Harvard study tracking 297 Claude Fable 5 users through the 19-day ban found that access to the model raised the share of pull requests merged within seven days by just 1.4 percentage points, a difference the paper calls not statistically significant. Across five Claude upgrades in 2026 covering 1,208 developers and more than 15,000 code submissions, no significant gain appeared in merge rates or weekly completed changes, even where benchmark scores rose 5 to 18 points. The authors suggest the tasks developers hand to agents already fall within what older models can do, so benchmark scores may not reflect real work productivity.
Heat for this event
Not enough continuous observations yet to draw a trend.