Skip to content
#Benchmarks
  1. The Decoder88

    Google releases Gemini 4 Hydrogen, scoring above the larger Argon four weeks after Argon's launch

    Google has released Gemini 4 Hydrogen, a fast general-purpose model for developers and everyday users, priced at $1.50 per million input tokens and $7.50 per million output tokens with a one-million-token context window. Google says Hydrogen scores 51 on the Artificial Analysis Intelligence Index at its default high thinking level and 54 at the new xhigh level, versus 50 for the larger Gemini 4 Argon. Independent testing by Artificial Analysis puts Hydrogen roughly on par with Claude Sonnet 5.5 at maximum thinking, but weaker on agentic terminal work, scoring 54 percent on Terminal-Bench 4.0 against 64 percent for Claude Sonnet 5.5.

    Why it matters: Google's cheaper Gemini 4 Hydrogen reportedly edges out the larger Argon on the Artificial Analysis Intelligence Index, while independent testing shows it trails Claude and GPT-6.1 on terminal tasks.

  2. AWS Machine Learning Blog60

    AWS open-sources UAEF, a framework-agnostic evaluation system for AI agents

    AWS has open-sourced UAEF (Universal Agent Evaluation Framework), a framework-agnostic system that uses adapters to normalize agent traces from LangGraph, LangChain, Strands Agents, Amazon Bedrock AgentCore and Langfuse into one canonical AgentTrace, then scores them with 36 built-in metrics across 7 dimensions. Results are stored per experiment in Amazon DynamoDB and Amazon S3, so teams can compare agent versions against a baseline, flag regressions with a default 5 percent threshold, and generate statistical reports plus LLM-written insights. The post says customers across 12 organizations in telecom, healthcare and financial services used UAEF before release, and the Apache 2.0 code, Python library, CDK deployment and per-framework notebooks are in the awslabs/uaef GitHub repository.

    Why it matters: The AWS team has open-sourced UAEF, which uses adapters to unify agent trajectories from different frameworks into a single format, then scores them across 7 dimensions using 36 metrics, providing a reusable evaluation layer for comparing agent versions across frameworks.

  1. OpenRouter Announcements60

    OpenRouter tests what fp8 and fp4 quantization do to LLM quality and price

    OpenRouter ran 280 exact-answer tasks against six endpoints of Qwen3.8 27B on October 20, 2026 and did not detect an accuracy difference between fp16, fp8 and fp4, with all six scoring between 98.3% and 99.2%. Across 1,455 endpoints, 59.5% report no precision at all and fp8 is the most common labeled format at 23.6%. Lower precision was cheaper for most models but not all, and provider choice moved price more than precision did, with the priciest same-precision endpoint a median 1.72 times the cheapest.

    Why it matters: OpenRouter measured fp8, fp4 and fp16 endpoints of one model across 240 exact-answer tasks and found no detectable accuracy gap, while price differences tracked the provider more than the precision.

  1. Latent Space85

    Anthropic releases Claude Fable 5.2 and Mythos 5.2, retaking the top LLM spot as its IPO roadshow begins

    Anthropic released Claude Fable 5.2 and Claude Mythos 5.2, which top most independent leaderboards, including Artificial Analysis Intelligence Index v5 at 65 and SemiAnalysis GDPx at 16.4%. Pricing drops 20% to $8/$40 per 1M input/output tokens, but SemiAnalysis reports cost per GDPx task rose from $67.16 to $88.15 because the model writes hundreds to thousands of test cases and spawns subagents to check its work. Multiple evals show the max effort level underperforming high, and Fable is capped at 50% of a weekly subscription limit.

    Why it matters: Anthropic releases Fable 5.2 and Mythos 5.2 with a 20% per-token price cut, yet independent evals show higher per-task cost and a max-effort setting that underperforms high.

  2. OpenRouter Announcements72

    OpenRouter benchmarks 15 vision models and Mistral OCR on scanned documents

    OpenRouter tested 15 vision models, Mistral OCR and Tesseract on eight self-made degraded scans, and Claude Opus 5.5 was most accurate with 262 of 264 key values correct at $16.42 per 1,000 pages, while Mistral OCR was best value at 258 of 264 for $2.02. On a faded receipt, GPT-6 Luna and GPT-6.1 Sol invented card digits, a server name and wifi passwords, errors that character error rates do not reveal. The post includes code to transcribe scans, turn reasoning off, cross-check two models from different labs and run Mistral OCR through the file-parser plugin.

    Why it matters: A hands-on benchmark of 15 vision models plus Mistral OCR and Tesseract on eight degraded scans, with runnable code for cross-checking two models to catch fabricated values.

  3. One Useful Thing78

    Ethan Mollick tests Claude Fable 5.2 across coding, business and research, calling it almost AGI

    2 on tasks spanning hardware hacking, business planning, humanities course building, academic research and a 450-world 3D portal project, and judged it almost AGI under his definition of matching most human experts at most computer-based intellectual tasks.

    Why it matters: Ethan Mollick's hands-on tests of Claude Fable 5.2 show strong general capability across coding, business and research, while exposing a persistent gap in self-judging when work is good enough.

  1. Artificial Analysis Articles82

    Gemini 4 Carbon tops Artificial Analysis Intelligence Index v5, ranks second on Coding Agent Index

    Google's Gemini 4 Carbon is the new No. 5 on the Artificial Analysis Coding Agent Index. 5 (max, 62), and it leads 4 of the 9 Index evaluations including HLE-Diamond and Terminal-Bench-Science. 1 rises from 4% to 11%; the model is being rolled out to selected users and is not yet publicly available.

    Why it matters: Gemini 4 Carbon tops Artificial Analysis Intelligence Index v5 at 59 and trails Claude Opus 5.5 by 3 points on the Coding Agent Index, with per-task cost and hallucination figures for comparison.

  1. Anthropic · Claude.dev72

    Anthropic adds build-eval and hillclimb commands to the claude-api skill in Claude Code

    Anthropic added two commands to the claude-api skill inside Claude Code: /claude-api build-eval builds an evaluation set in your codebase, and /claude-api hillclimb iterates on an application against that evaluation with a held-out set to catch overfitting.

    Why it matters: Anthropic adds build-eval and hillclimb commands to its claude-api skill, showing how Claude Code can design evaluations and iterate against them while guarding against overfitting.

  1. Modal Blog62

    Modal replicates Large Language Monkeys: LLaMA 3.1 8B matches GPT-4o on HumanEval via search

    Modal's Charles Frye replicated the core claim of the Large Language Monkeys paper on a new dataset, showing that repeated sampling lifts LLaMA 3.1 8B from 66.4% pass@1 to 90.5% pass@100 and 95.1% pass@1000 on HumanEval, versus GPT-4o's reported 90.2% pass@1. The runs used vLLM on Modal serverless GPUs and Modal Sandboxes for parallel test execution, cost well under $50, and showed smooth, predictable scaling across three orders of magnitude.

    Why it matters: A hands-on replication shows how repeated sampling with a small open model can approach GPT-4o on HumanEval, offering a concrete, low-cost recipe for trading compute for accuracy.

You are all caught up