Skip to content
Only Tutorials#Benchmarks
  1. LiveKit Blog62

    LiveKit tests Gemini 4 Hydrogen vs 3.8 Flash in a voice agent with deliberately broken tools

    LiveKit ran Gemini 4 Hydrogen and Gemini 3.8 Flash through the same Google plugin in a voice travel-concierge agent with deliberately failing tools, and Hydrogen made 0 false booking reports across 54 transcripts versus 2 for 3.8 Flash. With Hydrogen at high and 3.8 Flash at default, Hydrogen's first sentence arrived 1.2 s sooner (4.0 s vs 5.2 s), and at low on both it was 1.7 s sooner (2.8 s vs 4.6 s). The author recommends medium or high thinking for voice agents, noting xhigh's median 7.8 s to first sentence and a 52 s worst case.

    Why it matters: LiveKit's hands-on test shows Gemini 4 Hydrogen reporting every deliberately broken tool call and starting replies sooner than 3.8 Flash, with per-thinking-level latency data for voice agents.

  1. OpenRouter Announcements60

    OpenRouter tests what fp8 and fp4 quantization do to LLM quality and price

    OpenRouter ran 280 exact-answer tasks against six endpoints of Qwen3.8 27B on October 20, 2026 and did not detect an accuracy difference between fp16, fp8 and fp4, with all six scoring between 98.3% and 99.2%. Across 1,455 endpoints, 59.5% report no precision at all and fp8 is the most common labeled format at 23.6%. Lower precision was cheaper for most models but not all, and provider choice moved price more than precision did, with the priciest same-precision endpoint a median 1.72 times the cheapest.

    Why it matters: OpenRouter measured fp8, fp4 and fp16 endpoints of one model across 240 exact-answer tasks and found no detectable accuracy gap, while price differences tracked the provider more than the precision.

  1. OpenRouter Announcements72

    OpenRouter benchmarks 15 vision models and Mistral OCR on scanned documents

    OpenRouter tested 15 vision models, Mistral OCR and Tesseract on eight self-made degraded scans, and Claude Opus 5.5 was most accurate with 262 of 264 key values correct at $16.42 per 1,000 pages, while Mistral OCR was best value at 258 of 264 for $2.02. On a faded receipt, GPT-6 Luna and GPT-6.1 Sol invented card digits, a server name and wifi passwords, errors that character error rates do not reveal. The post includes code to transcribe scans, turn reasoning off, cross-check two models from different labs and run Mistral OCR through the file-parser plugin.

    Why it matters: A hands-on benchmark of 15 vision models plus Mistral OCR and Tesseract on eight degraded scans, with runnable code for cross-checking two models to catch fabricated values.

  1. Anthropic · Claude.dev72

    Anthropic adds build-eval and hillclimb commands to the claude-api skill in Claude Code

    Anthropic added two commands to the claude-api skill inside Claude Code: /claude-api build-eval builds an evaluation set in your codebase, and /claude-api hillclimb iterates on an application against that evaluation with a held-out set to catch overfitting.

    Why it matters: Anthropic adds build-eval and hillclimb commands to its claude-api skill, showing how Claude Code can design evaluations and iterate against them while guarding against overfitting.

  1. Modal Blog62

    Modal replicates Large Language Monkeys: LLaMA 3.1 8B matches GPT-4o on HumanEval via search

    Modal's Charles Frye replicated the core claim of the Large Language Monkeys paper on a new dataset, showing that repeated sampling lifts LLaMA 3.1 8B from 66.4% pass@1 to 90.5% pass@100 and 95.1% pass@1000 on HumanEval, versus GPT-4o's reported 90.2% pass@1. The runs used vLLM on Modal serverless GPUs and Modal Sandboxes for parallel test execution, cost well under $50, and showed smooth, predictable scaling across three orders of magnitude.

    Why it matters: A hands-on replication shows how repeated sampling with a small open model can approach GPT-4o on HumanEval, offering a concrete, low-cost recipe for trading compute for accuracy.

You are all caught up