How to Benchmark LLMs: A Code-Graded Guide from OpenRouter

OpenRouter published a guide for benchmarking LLMs on your own application's inputs, grading outputs with code and comparing accuracy, billed cost and request time. On its 60 synthetic tasks, GPT-6.1 Sol passed 180/180 trials across three repeats, while DeepSeek had the lowest response cost at 157/180 passes. Agreement with the public ranking was weak, with a Spearman correlation of 0.109.