OpenRouter Publishes Code-Graded LLM Benchmarking Guide
Start here
OpenRouter published a guide for benchmarking LLMs on a user's own application inputs, grading outputs with code and comparing accuracy, billed cost and request time. On its 60 synthetic tasks, GPT-6.1 Sol passed 180 of 180 trials across three repeats, while DeepSeek had the lowest response cost at 157 of 180 passes. OpenRouter reports that agreement with the public ranking was weak, with a Spearman correlation of 0.109. The teaching runner, JSON example and comparisons passed offline checks on Windows x64 and Linux; the bundled full-study sandbox and native Ori setup support Windows x64 only. Offline checks make no API calls, and paid runs require the explicit —live flag.
Generated from reports · updated 13 h ago
Report timeline
Follow the reports to see the event from different sides.
- OpenRouter AnnouncementsHow to Benchmark LLMs: A Code-Graded Guide from OpenRouter
OpenRouter published a guide for benchmarking LLMs on your own application's inputs, grading outputs with code and comparing accuracy, billed cost and request time. On its 60 synthetic tasks, GPT-6.1 Sol passed 180/180 trials across three repeats, while DeepSeek had the lowest response cost at 157/180 passes. Agreement with the public ranking was weak, with a Spearman correlation of 0.109.
Heat for this event
Not enough continuous observations yet to draw a trend.