Skip to content
Only First-party#Benchmarks
November 13Friday
  1. LiveKit Blog62

    LiveKit tests Gemini 4 Hydrogen vs 3.8 Flash in a voice agent with deliberately broken tools

    LiveKit ran Gemini 4 Hydrogen and Gemini 3.8 Flash through the same Google plugin in a voice travel-concierge agent with deliberately failing tools, and Hydrogen made 0 false booking reports across 54 transcripts versus 2 for 3.8 Flash. With Hydrogen at high and 3.8 Flash at default, Hydrogen's first sentence arrived 1.2 s sooner (4.0 s vs 5.2 s), and at low on both it was 1.7 s sooner (2.8 s vs 4.6 s). The author recommends medium or high thinking for voice agents, noting xhigh's median 7.8 s to first sentence and a 52 s worst case.

    Why it matters: LiveKit's hands-on test shows Gemini 4 Hydrogen reporting every deliberately broken tool call and starting replies sooner than 3.8 Flash, with per-thinking-level latency data for voice agents.

  2. OpenRouter Announcements58

    OpenRouter tests 11 rerankers on a 274-question RAG benchmark

    OpenRouter benchmarked 11 rerankers on a 274-question synthetic RAG dataset and found that reranking the top 30 vector-search candidates with voyageai/rerank-3-lite raised precision@1 from 81.0% to 97.1% at about $0.10 per 1,000 queries. The guide recommends retrieving roughly 30 chunks, reranking them, and sending only the top 3 to the LLM, which in its test lifted answer accuracy to 99.3% while cutting answering cost from $0.22 to $0.16 per 1,000 questions. It also flags failure modes: omitting the input_type parameter cost 7 to 12 points of precision@1, and cohere/rerank-v3.5 silently ignores content beyond its 4,096-token limit.

  3. LangChain Blog45

    What Is Eval-Driven Development? How to Improve AI Agents

    LangChain explains eval-driven development (EDD), a three-step process for building AI agents: define what good looks like, write the eval before you build, and compare results after each change. Because agents are non-deterministic, evals score runs across a dataset of examples and can use LLM-as-a-judge, with real agent runs in LangSmith as a source of test inputs.

  4. AWS Machine Learning Blog38

    Amazon Bedrock AgentCore Evaluations: Built-in, DeepEval and AutoEval evaluators

    Amazon Bedrock AgentCore Evaluations now lets teams run AWS built-in evaluators alongside managed DeepEval and AutoEval metrics, plus custom evaluators derived from either. The built-in catalog spans 15 evaluators across response quality, goal completion, tool use, skills and safety, with three trajectory checks for expected tool paths. DeepEval adds multi-turn metrics such as ConversationCompleteness and KnowledgeRetention, while AutoEval contributes specialized signals including security.

  5. AWS Machine Learning Blog60

    AWS open-sources UAEF, a framework-agnostic evaluation system for AI agents

    AWS has open-sourced UAEF (Universal Agent Evaluation Framework), a framework-agnostic system that uses adapters to normalize agent traces from LangGraph, LangChain, Strands Agents, Amazon Bedrock AgentCore and Langfuse into one canonical AgentTrace, then scores them with 36 built-in metrics across 7 dimensions. Results are stored per experiment in Amazon DynamoDB and Amazon S3, so teams can compare agent versions against a baseline, flag regressions with a default 5 percent threshold, and generate statistical reports plus LLM-written insights. The post says customers across 12 organizations in telecom, healthcare and financial services used UAEF before release, and the Apache 2.0 code, Python library, CDK deployment and per-framework notebooks are in the awslabs/uaef GitHub repository.

    Why it matters: The AWS team has open-sourced UAEF, which uses adapters to unify agent trajectories from different frameworks into a single format, then scores them across 7 dimensions using 36 metrics, providing a reusable evaluation layer for comparing agent versions across frameworks.

  6. AWS Machine Learning Blog40

    Agent DLC: Engineering AI Agents from Prototype to Production

    AWS introduces the Agent Development Lifecycle (Agent DLC), a framework with one lifecycle, two loops, and a single explicit standard for judging every agent change. It addresses three agent properties that break conventional CI/CD: non-determinism, prompts as source code, and independently moving dependencies. A reference HR policy assistant shows chunk-level retrieval precision of 1.00 across all three test questions while fact-level precision fell to 0.09 and retrieval coverage hit 0.00 on one question.

November 12Thursday
  1. OpenRouter Announcements28

    How to Benchmark LLMs: A Code-Graded Guide from OpenRouter

    OpenRouter published a guide for benchmarking LLMs on your own application's inputs, grading outputs with code and comparing accuracy, billed cost and request time. On its 60 synthetic tasks, GPT-6.1 Sol passed 180/180 trials across three repeats, while DeepSeek had the lowest response cost at 157/180 passes. Agreement with the public ranking was weak, with a Spearman correlation of 0.109.

November 11Wednesday
  1. inclusionAI Models34

    inclusionAI Releases LemonDecider 4B v3 Decision Model Based on Qwen3.5-4B

    inclusionAI released LemonDecider 4B v3, a decision model fine-tuned from Qwen3.5-4B that returns choice, yes/no or ordinal probabilities via a single joint forward pass instead of generating answer tokens. It was trained on about 1.20M decision/QA rows and scored 61.67 on the public Decision Index 0.4 suite, with median request latency of 49.9 ms.

  2. OpenRouter Announcements60

    OpenRouter tests what fp8 and fp4 quantization do to LLM quality and price

    OpenRouter ran 280 exact-answer tasks against six endpoints of Qwen3.8 27B on October 20, 2026 and did not detect an accuracy difference between fp16, fp8 and fp4, with all six scoring between 98.3% and 99.2%. Across 1,455 endpoints, 59.5% report no precision at all and fp8 is the most common labeled format at 23.6%. Lower precision was cheaper for most models but not all, and provider choice moved price more than precision did, with the priciest same-precision endpoint a median 1.72 times the cheapest.

    Why it matters: OpenRouter measured fp8, fp4 and fp16 endpoints of one model across 240 exact-answer tasks and found no detectable accuracy gap, while price differences tracked the provider more than the precision.

  4. inclusionAI Models40

    inclusionAI Releases ZoomGain-9B and ZoomGain-4B Models with ZoomGainBench and RL Training Data

    inclusionAI released the ZoomGain-9B and ZoomGain-4B vision-language model checkpoints, alongside the ZoomGainBench evaluation benchmark and ZoomGain RL training data. ZoomGain mines global-to-zoom perception gaps to build fine-grained visual supervision, and ZoomGain-9B reached superior performance on multiple fine-grained and general perception benchmarks. ZoomGainBench contains 1,000 questions over 941 images, while the RL dataset holds 9,000 training records and 7,713 source images.

  5. inclusionAI Models40

    inclusionAI Releases SingVL S1 4B Multimodal Decision Model on Qwen3.5

    inclusionAI released SingVL S1, a 4B multimodal decision model built on Qwen3.5 that answers named questions over text or JSON and up to four images in a single Transformers forward pass. It supports choice, yes/no (noul) and 2–10 level score questions, scoring 73.94 on the Public Vision Index and 69.76 on the Public Text Index. The model ships with Python inference code and a FastAPI HTTP server.

November 10Tuesday
  1. OpenRouter Announcements72

    OpenRouter benchmarks 15 vision models and Mistral OCR on scanned documents

    OpenRouter tested 15 vision models, Mistral OCR and Tesseract on eight self-made degraded scans, and Claude Opus 5.5 was most accurate with 262 of 264 key values correct at $16.42 per 1,000 pages, while Mistral OCR was best value at 258 of 264 for $2.02. On a faded receipt, GPT-6 Luna and GPT-6.1 Sol invented card digits, a server name and wifi passwords, errors that character error rates do not reveal. The post includes code to transcribe scans, turn reasoning off, cross-check two models from different labs and run Mistral OCR through the file-parser plugin.

    Why it matters: A hands-on benchmark of 15 vision models plus Mistral OCR and Tesseract on eight degraded scans, with runnable code for cross-checking two models to catch fabricated values.

  2. Apple Machine Learning Research31

    Apple study evaluates LLM judges for personalized writing style assessment

    Apple researchers tested how well LLM judges assess whether generated text matches a user's preferred writing style, generating text with four LLMs across three long-form writing tasks and comparing judge annotations against human pairwise labels. Judge quality correlated only weakly with general LLM ability on MMLU (Pearson's ρ ≤ 0.2) and varied by writing task and evaluation criteria, and paired comparisons proved more consistent than independent scoring. The strongest LLMs did not most reliably reproduce human evaluations, possibly due to exaggerated sensitivity to subtle details.

  4. Recraft Blog22

    Best AI Meme Generators in 2026: 7 Tools Compared

    Seven AI meme generators — Imgflip, Canva, Supermeme.ai, Recraft, Kapwing, ChatGPT and Magic Hour — were tested on free tier, watermark, templates, caption rendering, export sizes and API using the same three meme briefs. The comparison finds no universal winner: Imgflip wins for classic templates, Supermeme.ai for idea-to-caption, Recraft and ChatGPT for original images, Kapwing for video and GIF, and Canva when the meme must fit a layout.

November 9Monday
  1. Recraft Blog22

    Best AI Sticker Generators in 2026: 7 Tools Compared

    Recraft is the best AI sticker generator for die-cut outlines, transparent PNGs, and SVGs from one canvas, while Canva is better for sticker sheets and print orders in the same editor. Seven tools were scored on November 9, 2026 against five print criteria, and only Recraft, Kittl, and Canva Print offer a real path to cut lines or print sheets. Firefly, Ideogram, Leonardo AI, and ChatGPT win narrower jobs such as photo-to-sticker conversion or chat packs.

  2. Recraft Blog38

    Best AI Character Generators in 2026: Recraft, Midjourney, Gemini, ChatGPT and More Compared

    A comparison of seven AI character generators scored on pose consistency, reference images, style control, export formats and free tiers found no single best tool for every brief. Recraft leads for keeping one character across poses with reusable styles and SVG vector export, while Midjourney wins single-frame looks but has no free plan, with Basic at $10 per month. Gemini and ChatGPT suit iterating a character inside a chat, and Recraft, Gemini, ChatGPT, Leonardo, Ideogram and OpenArt all offer a free or weekly-credit tier.

November 6Friday
  1. Modal Blog22

    How Snorkel runs realistic agent simulations with Modal

    Snorkel uses Modal's GPU and CPU sandboxes to run millions of agent simulations per month, peaking at tens of thousands of sandboxes an hour, and relies on Modal for inference on the open-weight models it evaluates. Adopting Modal's memory bursting cut Snorkel's total sandbox bill by around 20%, with roughly 10-15% of its sandboxes using bursting today. Modal was the only sandbox provider able to meet Snorkel's GPU sandbox needs at scale.

November 3Tuesday
  1. Ai2 · Allen Institute for AI55

    Ai2 releases StateBench to test how language models track state changes across long documents

    Ai2 introduced StateBench, a benchmark with four difficulty levels for long-context state tracking, from immutable assignment to self-referential and cross-referential updates. In tests, Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna handled the easier levels but dropped to 13% or lower on the longest general finite problems, while 275M-parameter Gated DeltaNet and hybrid models trained from scratch generalized almost perfectly on the first three levels and transformers never learned the hardest one.

  2. OpenAI Alignment Blog58

    OpenAI examines eval-production behavior gaps in agent safety and the role of eval-awareness

    OpenAI researchers measured an eval-production behavior gap in agent safety: across six served models, success on the Respecting Warnings evaluation was 10.2 to 37.4 percentage points lower than on matched Codex production decisions, with the largest gaps in GPT-5.6 Sol and GPT-6 Sol. General eval-awareness alone showed weak association with the gap (R² = 0.008), while specific eval-objective awareness was stronger (R² = 0.370), and models most often verbalizing the wrong objective (resisting prompt injection) had the biggest gaps. In an intervention on GPT-6.1 Sol, a generic evaluation cue left success nearly unchanged (63.1% vs 63.4%), the correct objective cue raised it to 91.6%, and the incorrect objective cue cut it to 17.6%.

October 30Friday
  1. Microsoft Research40

    Microsoft builds synthetic enterprise data platform to evaluate Copilot

    Microsoft's Industry Evaluation Data Generation Platform creates full-scale synthetic enterprise environments to evaluate Microsoft Copilot without using customer data. Grounded in public sources such as O*NET occupational data, job postings, and SEC filings, it generates realistic tenants with users, meetings, emails, and documents, plus automated query pipelines and quality frameworks. Microsoft says this establishes the first standardized evaluation baseline for Copilot.

October 28Wednesday
October 27Tuesday
  1. Ai2 · Allen Institute for AI58

    Ai2 study tests whether LLMs can reliably judge research idea novelty

    Ai2 released Old Ideas, Novel Problems, a study and benchmark testing how reliably LLMs evaluate the novelty of research ideas, along with data, code and a pipeline for building new evaluation sets. Across six judges including GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6, small prompt changes shifted accuracy by more than 50 percentage points on AI-generated ideas, and a simple prompted Claude Opus 4.6 baseline scored 0.854 macro-F1 at $2.59 versus 0.642 at $30.66 for Idea Novelty Checker and 0.772 at $82.05 for Sakana AI's AI Scientist.

October 21Wednesday
  1. Amazon Science38

    Amazon Taps AVERI for Independent Review of Its AI Safety Evaluation Datasets

    Amazon partnered with independent nonprofit AVERI to review the quality of its internal safety evaluation datasets, not just model outputs. AVERI validated topical coverage, prompt diversity, and low contamination risk, while recommending broader multilingual coverage, more multistep adversarial prompts, and clearer grading criteria. Amazon responded with refreshed test sets, tiered difficulty levels, and a quarterly review cadence.

  2. OpenAI Alignment Blog28

    OpenAI proposes pair-matched and post-control metrics to extend g-mean² monitorability

    OpenAI's Alignment Blog adds two metrics to g-mean², its dataset-level measure of monitor discrimination. A pair-matched metric compares models only on task instances that support the relevant positive, negative or intervention-valid comparison, while a post-control integration metric simulates monitor-gated rejection and estimates the probability that the deployed system still emits a bad output. The post also derives a pessimistic lower bound for intervention evaluations.

October 19Monday
  1. inclusionAI Repositories58

    inclusionAI releases Ming-3.0-tts speech synthesis models in 0.5B dense and 7.9B MoE tiny variants

    inclusionAI released Ming-3.0-tts, an open-weight multilingual speech synthesis family with two models: Ming-3.0-tts-0.5b, a 0.5B dense model, and Ming-3.0-tts-tiny, a MoE model built on Ling V3 with 7.9B total and 1.3B active parameters. Both ship with inference code, a task cookbook and benchmark tables, and support voice cloning, text-based voice design, free-form natural-language delivery control, 14 inline paralinguistic tags, emotion and speed/pitch/volume control, plus Chinese, English, German, Spanish, French, Korean, Japanese, Sichuanese and Cantonese. On the October 2026 Artificial Analysis text-to-speech leaderboards, Ming-3.0-tts ranks 1st among open-weights models on Controlled Voice, 1st overall on Pronunciation Robustness at 93.2%, and 3rd among open-weights models on Provider Voice.

  2. Modal Blog38

    How Legora runs 2,000 parallel agent evals on Modal Sandboxes

    Legora now runs each of its legal agent evals inside a separate Modal Sandbox, booting its full application pinned to a specific commit so every run reflects the exact product version. A typical experiment is 100 cases, 4 harness variants, and 5 trials each — 2,000 runs in parallel — with Modal Volumes snapshotting document, database, and permission state. Legora also moved from gVisor to Modal VM Sandboxes, removing its Docker networking and FUSE workarounds, and cut agent cost by 30% via smarter compaction and custom tool search.

October 6Tuesday
  1. LiveKit Blog38

    Retell vs. Vapi vs. LiveKit: A Developer Comparison for Voice AI Agents

    LiveKit published a developer-focused comparison of Retell, Vapi and LiveKit for building voice AI agents, framing the choice around four questions: who iterates on the agent after launch, what happens when needs fall outside defaults, whether it must run beyond phone calls, and cost at scale. Retell is a managed platform with a visual editor, Vapi a developer-oriented managed platform configured via API and dashboard, and LiveKit an Apache 2.0 open source framework with self-hosting. Estimated all-in cost per minute runs about $0.070–$0.310 for Retell, $0.070–$0.116 for Vapi and $0.037–$0.099 for LiveKit.

October 5Monday
  1. GitHub Blog · AI & ML58

    GitHub releases ReviewBench, an open benchmark for AI code review

    GitHub released ReviewBench, an open benchmark for AI code review agents built on 219 public pull requests across 19 languages, with language and repository-size distributions modeled on 103.9 million GitHub pull requests. The golden set draws findings from human reviewers, author follow-up commits, static analysis and multiple frontier LLMs, graded by Claude Sonnet 5 under a published rubric, and senior engineers independently re-labeled every finding with 96.6% agreement. GitHub says its offline evaluation of Copilot code review anticipated the direction of a production A/B test, where addressed rate rose 8.0%, recall rose 13.6% and cost per review fell 8.0%.

September 28Monday
  1. Anthropic · Claude.dev72

    Anthropic adds build-eval and hillclimb commands to the claude-api skill in Claude Code

    Anthropic added two commands to the claude-api skill inside Claude Code: /claude-api build-eval builds an evaluation set in your codebase, and /claude-api hillclimb iterates on an application against that evaluation with a held-out set to catch overfitting.

    Why it matters: Anthropic adds build-eval and hillclimb commands to its claude-api skill, showing how Claude Code can design evaluations and iterate against them while guarding against overfitting.

August 18Tuesday
  1. Vapi Blog55

    Vapi launches Simulations to test voice agents against AI callers

    Vapi launched Simulations, a native AI-powered testing feature that places AI testers with distinct personalities and customer-derived scenarios on the other end of a call with a voice agent. Each run returns pass/fail results, a full transcript and a recording, and teams can mock API responses, run cheap text-based chat mode or full voice mode, and trigger runs via API with quality gates that block deploys when pass rates drop.

August 3Monday
  1. Vapi Blog33

    Why model flexibility matters in voice: Speechify Simba 3.2 and Fish Audio S2.1-Pro top Vapi's Humanness Index

    Speechify's Simba 3.2 debuted atop Vapi's Humanness Index at 99, one point under the human baseline of 100, and Fish Audio's S2.1-Pro entered at 94 with 141 ms time to first audio. The prior leader, xAI's Grok TTS, held the top spot for only three weeks at 95, and the leading score has since moved from 95 to 99. Vapi argues that model-agnostic voice platforms let teams swap TTS models as rankings shift.

July 13Monday
  1. Vapi Blog58

    Vapi Humanness Index: Grok TTS scores 95 and MiniMax Speech 2.5 scores 93 against a real human at 100

    Vapi's Humanness Index, a blind crowdsourced leaderboard where listeners pick which of two cloned voices sounds more human, places xAI's Grok TTS at 95 and MiniMax Speech 2.5 at 93 against a real human recording scored at 100 after more than 11,000 votes. Every battle uses the same cloned voice and script across models, and the real human recording is included in the comparison set, so a 95 means listeners chose Grok TTS at nearly the same rate as the human on the same quote. Vapi argues naturalness is no longer the bottleneck for voice agents, since callers experience the voice rather than the reasoning behind it.

April 29Wednesday
  1. Factory58

    Factory benchmarks 13 models on code review, finds GPT-5.2 best and MiniMax M2.7 cheapest

    Factory built an open-source benchmark testing 13 models on 50 real pull requests from Sentry, Grafana, Keycloak, Discourse and Cal.com, with each model reviewing every PR at least three times at high reasoning effort and an LLM judge scoring findings against a human-curated golden set. GPT-5.2 led at 60.5% F1 for $1.25 per PR, ahead of Claude Opus 4.6 at 59.8% F1 and $3.11 per PR, while MiniMax M2.7 reached 45.6% F1 at $0.15 per PR and Kimi K2.5 hit 51.9% F1 at $0.41. Factory says cost explains only about 21% of quality variance, and it open-sourced the golden set, evaluation scripts and a Droid Action GitHub Action supporting all 13 models.

January 20Tuesday
  1. Factory58

    Factory launches Agent Readiness to score how well codebases support AI coding agents

    Factory launched Agent Readiness, which evaluates how well a codebase supports autonomous development across eight technical pillars and five maturity levels, runnable via /readiness-report in Droid, an organization dashboard, or an API. Each criterion is binary and a level unlocks only when 80% of its criteria and all prior levels pass; grounding each evaluation on the repository's previous report cut score variance from an average of 7% to 0.6% across 9 benchmark repositories over six weeks. Reports can trigger an agent that opens a pull request to fix failing criteria such as missing AGENTS.md, linter configuration, or pre-commit hooks.

July 2, 2025Wednesday
  1. Modal Blog40

    How Modal Used Evals and Inference-Time Compute Scaling to Generate Scannable QArt Codes

    Modal used evals and inference-time compute scaling to improve its QArt code system, boosting QR scan rate to its 95% service-level objective while improving aesthetic quality and returning codes in under 20s at p95. The work follows the standard ML playbook of collecting human judgments, aligning to them, then scaling, addressing the "it worked on my prompt" gap that leaves many generated QR codes unscannable.

June 18, 2025Wednesday
  1. Vapi Blog22

    Claude vs ChatGPT: The Complete Comparison Guide

    Claude and ChatGPT are compared across conversational quality, context memory, speed, features, creative writing, research, pricing, and interface. Claude handles up to 200,000+ tokens for Pro users and excels at creative writing and nuanced conversation, while ChatGPT supports 128,000 tokens, responds in 2-3 seconds versus Claude's 5-10 seconds, and offers image generation and Custom GPTs. Both Pro tiers cost $20 per month, with Claude Max at $100-$200 and ChatGPT Pro at $200.

June 5, 2025Thursday
  1. Vapi Blog22

    GPT-4.1 vs Claude 3.7 Sonnet: Which AI Delivers Better Voice Agents?

    Vapi compares GPT-4.1 and Claude 3.7 Sonnet as the LLM behind voice agents, weighing conversation quality, context handling, reasoning transparency, platform integration and cost. GPT-4.1 offers a 1,000,000-token context window and 32,768-token outputs with 75% caching discounts, while Claude 3.7 provides a 200,000-token context and 128,000-token outputs. Both models are selectable on Vapi's voice agent platform.

May 30, 2025Friday
  1. Vapi Blog15

    Best Speechify Alternatives: 5 TTS Tools Compared

    A comparison of five Speechify alternatives picks NaturalReader for cross-device use and app integrations at $119 per year, Voice Dream Reader for iPhone voice control at a $14.99 one-time price, and Murf AI for professional voices from about $19 per month. ReadSpeaker focuses on reading web pages across 45 languages from $4.90 per month, while Balabolka is a free Windows-only option.