Surge AI benchmark: smallest Claude beats largest GPT on engineering judgment
Start here
Surge AI evaluated five frontier models on sudo L7, its benchmark of staff-level engineering behavior using real licensed codebases, and reported that every Claude model outscored the top GPT models, with the smallest Claude (Haiku) beating both Astra and Sol. The two model families performed similarly on functional correctness, but Claude pulled ahead on flagging flawed assumptions, hidden risks and the consequences of changes, such as caching customer balances or a Terraform typo that would destroy and recreate infrastructure. According to Surge AI, Claude's final reports were roughly 10x longer. GPT made fewer inaccurate claims in total, but produced about 4x as many inaccurate statements per word. On cost, Sol came to $0.91 per task versus $1.68 for Haiku.
Generated from reports · updated 45 min ago
Report timeline
Follow the reports to see the event from different sides.
- Hacker News · AISurge AI benchmark finds smallest Claude outperforms largest GPT on software engineering judgment
Surge AI evaluated five frontier models on sudo L7, its benchmark of staff-level engineering behavior using real licensed codebases, and every Claude model outscored the top GPTs, with Haiku beating Astra and Sol. The two families performed similarly on functional correctness, but Claude pulled ahead on flagging flawed assumptions, hidden risks and consequences of changes, such as caching customer balances or a Terraform typo that would destroy and recreate infrastructure. Claude's final reports were roughly 10x longer, and while GPT made fewer inaccurate claims in total, it produced about 4x as many inaccurate statements per word; Sol cost $0.91 per task versus Haiku's $1.68.
Heat for this event
Not enough continuous observations yet to draw a trend.