Skip to content
Hot eventDeveloping

AI audit of 158,059 astrophysics papers finds 872 errors

1 reports1 reporting sourcesUpdated 20 min ago

Start here

AI digest

Casey Handmer, a Caltech-trained theoretical astrophysicist, used AI to screen 158,059 papers and 554,670 citations in his subfield, then had reviewers recompute 659 of the most-cited papers; they verified 872 errors across 433 papers. Of those errors, 367 papers copied one, only 47 were ever criticised in print, and about half of the copied errors were still being copied seven years after publication. Handmer says the audit would cost about $2,720 in model time at list price, roughly $3 per verified error, though in practice it ran within a Claude Max subscription; the whole project took about 4.9 billion tokens and about 44 hours of his time over 41 days. The deep check has so far reached only 4% of the highly cited papers on the map.

Generated from reports · updated 15 min ago

Report timeline

Follow the reports to see the event from different sides.

November 18
  1. Hacker News · AI
    Astrophysicist uses AI to audit 158,059 papers in his field and finds 872 errors

    Casey Handmer, a Caltech-trained theoretical astrophysicist, used AI to screen 158,059 papers and 554,670 citations in his subfield, then had reviewers recompute 659 of the most-cited papers and verify 872 errors across 433 papers. Of those errors, 367 papers copied one, only 47 were ever criticised in print, and about half of copied errors were still being copied seven years after publication. The project cost roughly $2,720 in model time at list price, about $3 per verified error, and the deep check has so far reached only 4% of the highly cited papers on the map.

Heat for this event

Not enough continuous observations yet to draw a trend.