Surge AI benchmark finds smallest Claude outperforms largest GPT on software engineering judgment
The Smallest Claude Is a Better Software Engineer Than the Biggest GPT
Surge AI evaluated five frontier models on sudo L7, its benchmark of staff-level engineering behavior using real licensed codebases, and every Claude model outscored the top GPTs, with Haiku beating Astra and Sol. The two families performed similarly on functional correctness, but Claude pulled ahead on flagging flawed assumptions, hidden risks and consequences of changes, such as caching customer balances or a Terraform typo that would destroy and recreate infrastructure. Claude's final reports were roughly 10x longer, and while GPT made fewer inaccurate claims in total, it produced about 4x as many inaccurate statements per word; Sol cost $0.91 per task versus Haiku's $1.68.
Source:Hacker News · AI · surgehq.ai