Skip to content
Hot eventDeveloping

Surge AI benchmark: smallest Claude beats largest GPT on engineering judgment

1 reports1 reporting sourcesUpdated 1 h ago

Start here

AI digest

Surge AI evaluated five frontier models on sudo L7, its benchmark of staff-level engineering behavior using real licensed codebases, and reported that every Claude model outscored the top GPT models, with the smallest Claude (Haiku) beating both Astra and Sol. The two model families performed similarly on functional correctness, but Claude pulled ahead on flagging flawed assumptions, hidden risks and the consequences of changes, such as caching customer balances or a Terraform typo that would destroy and recreate infrastructure. According to Surge AI, Claude's final reports were roughly 10x longer. GPT made fewer inaccurate claims in total, but produced about 4x as many inaccurate statements per word. On cost, Sol came to $0.91 per task versus $1.68 for Haiku.

Generated from reports · updated 45 min ago

Report timeline

Follow the reports to see the event from different sides.

November 17
  1. Hacker News · AI
    Surge AI benchmark finds smallest Claude outperforms largest GPT on software engineering judgment

    Surge AI evaluated five frontier models on sudo L7, its benchmark of staff-level engineering behavior using real licensed codebases, and every Claude model outscored the top GPTs, with Haiku beating Astra and Sol. The two families performed similarly on functional correctness, but Claude pulled ahead on flagging flawed assumptions, hidden risks and consequences of changes, such as caching customer balances or a Terraform typo that would destroy and recreate infrastructure. Claude's final reports were roughly 10x longer, and while GPT made fewer inaccurate claims in total, it produced about 4x as many inaccurate statements per word; Sol cost $0.91 per task versus Haiku's $1.68.

Heat for this event

Not enough continuous observations yet to draw a trend.