Skip to content
#Benchmarks
November 9Monday
  1. Recraft Blog22

    Best AI Sticker Generators in 2026: 7 Tools Compared

    Recraft is the best AI sticker generator for die-cut outlines, transparent PNGs, and SVGs from one canvas, while Canva is better for sticker sheets and print orders in the same editor. Seven tools were scored on November 9, 2026 against five print criteria, and only Recraft, Kittl, and Canva Print offer a real path to cut lines or print sheets. Firefly, Ideogram, Leonardo AI, and ChatGPT win narrower jobs such as photo-to-sticker conversion or chat packs.

  2. Recraft Blog38

    Best AI Character Generators in 2026: Recraft, Midjourney, Gemini, ChatGPT and More Compared

    A comparison of seven AI character generators scored on pose consistency, reference images, style control, export formats and free tiers found no single best tool for every brief. Recraft leads for keeping one character across poses with reusable styles and SVG vector export, while Midjourney wins single-frame looks but has no free plan, with Basic at $10 per month. Gemini and ChatGPT suit iterating a character inside a chat, and Recraft, Gemini, ChatGPT, Leonardo, Ideogram and OpenArt all offer a free or weekly-credit tier.

  3. Artificial Analysis Articles71

    Meta's Muse Ember is the cheapest model scoring 36 on the Artificial Analysis Intelligence Index

    Meta's Muse Ember scores 36 on the Artificial Analysis Intelligence Index, matching Claude Haiku 5.5 (max) at $0.14 per task versus $0.41 for Claude Haiku 5.5. It stands out on AutomationBench-AA at 68.1% and scores 46 on the Coding Agent Index in Muse Code, ahead of Claude Haiku 5.5 in Claude Code (37) and GPT-6 Luna in Codex (31). Its main weakness is knowledge: AA-Omniscience accuracy of 21.1% versus 34.7% for Claude Haiku 5.5, largely because it declines to answer 64% of questions, which holds its hallucination rate to 17.4%.

  4. Artificial Analysis Articles58

    Artificial Analysis releases Endpoint Accuracy Index v1.1 with tighter long-context intervals and six-model coverage

    Artificial Analysis launched Endpoint Accuracy Index v1.1, which tightens AA-LCR-25 confidence intervals and adds coverage for GLM-5.3 Flash and Muse Spark Open. Across 103 serverless endpoints benchmarked, 24 score statistically below the self-hosted reference, with causes including output token caps, reduced reasoning, tool call formatting errors and request timeouts. Raising AA-LCR-25 to 25 repeats per question narrows its average 95% confidence interval from ±4.7 to ±2.6 points, and the composite index interval from ±5.5 to ±4.3 points across the 45 endpoints scored under both versions.

November 8Sunday
  1. Artificial Analysis Articles38

    Artificial Analysis Decision Index v2 Released with New Decision Score and Private Benchmarks

    Artificial Analysis released Decision Index v2, with 45% of the Index now built on private datasets, up from 25% in v1. Drex Max v1.1 from Nace AI narrowly leads decision models at 73, within 1 point of GLiDE v1.2 and Mammoth 27B. The new Decision Score rewards both correctness and calibrated confidence, while AA-Paperclip and AA-Symmetry add private document and consistency tests.

November 6Friday
  1. SemiAnalysis75

    SemiAnalysis benchmarks Google TPUv7 Ironwood on agentic inference against NVIDIA GPUs

    SemiAnalysis released the first long-context, multi-turn agentic inference results on InferenceX/AgentX for Google TPUv7 Ironwood across four models, finding it competitive on performance per dollar or state-of-the-art in several cases. On GLM-5.2 in FP8, TPUv7 delivered 1.2x to 2.7x the tokens per dollar of B200, B300 and MI355X from 60 to 300 tok/s/user, while FP4 GPUs still led on GLM-5.3 because TPUv7 lacks native FP4 support until TPUv8 in Q1 2027. The piece also argues NVIDIA's bureaucracy slows driver and ecosystem contributions compared with AMD and Google.

  2. Artificial Analysis Articles40

    VIALS-AA: Different Paths to Scientific Image Reasoning

    VIALS-AA analyzes how 19 models reason over scientific images, classifying 9,177 rollouts of tool calls. Viewing images accounted for 38.3% of calls and image preparation or measurement 47.3%, while image measurement was just 5–6% of calls for GPT-6 Astra (max) and GPT-6.1 Sol (max) versus 25–27% for Claude Opus 5.5 (max) and Claude Sonnet 5.5 (max). Gemini 3.8 Flash (high) produced 86.7% of the cohort's commentary/no-op calls, and cost per task varies by nearly 100× across models.

  3. Modal Blog22

    How Snorkel runs realistic agent simulations with Modal

    Snorkel uses Modal's GPU and CPU sandboxes to run millions of agent simulations per month, peaking at tens of thousands of sandboxes an hour, and relies on Modal for inference on the open-weight models it evaluates. Adopting Modal's memory bursting cut Snorkel's total sandbox bill by around 20%, with roughly 10-15% of its sandboxes using bursting today. Modal was the only sandbox provider able to meet Snorkel's GPU sandbox needs at scale.

November 5Thursday
  1. Artificial Analysis Articles58

    Nanbeige releases Nanbeige4.3-33B, scoring 20 on the Artificial Analysis Intelligence Index

    Nanbeige has released Nanbeige4.3-33B, an open-weights dense reasoning model under Apache 2.0 that scores 20 on the Artificial Analysis Intelligence Index v5, matching Qwen3.8 27B (xhigh) at 20. The 33B model has a 524K-token context window, text-only input and output, and no confirmed public APIs, and it scores 30% on Terminal-Bench 4.0 versus 6% for Qwen3.8 27B while trailing on HLE-Diamond (7% vs. 15%). On AA-Omniscience v1.1 it records an 18% hallucination rate versus 29% for Qwen3.8 27B, with a 71% decline rate, and it uses 158K output tokens per Intelligence Index task versus 110K for Qwen3.8 27B.

November 3Tuesday
  1. Ai2 · Allen Institute for AI55

    Ai2 releases StateBench to test how language models track state changes across long documents

    Ai2 introduced StateBench, a benchmark with four difficulty levels for long-context state tracking, from immutable assignment to self-referential and cross-referential updates. In tests, Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna handled the easier levels but dropped to 13% or lower on the longest general finite problems, while 275M-parameter Gated DeltaNet and hybrid models trained from scratch generalized almost perfectly on the first three levels and transformers never learned the hardest one.

  2. OpenAI Alignment Blog58

    OpenAI examines eval-production behavior gaps in agent safety and the role of eval-awareness

    OpenAI researchers measured an eval-production behavior gap in agent safety: across six served models, success on the Respecting Warnings evaluation was 10.2 to 37.4 percentage points lower than on matched Codex production decisions, with the largest gaps in GPT-5.6 Sol and GPT-6 Sol. General eval-awareness alone showed weak association with the gap (R² = 0.008), while specific eval-objective awareness was stronger (R² = 0.370), and models most often verbalizing the wrong objective (resisting prompt injection) had the biggest gaps. In an intervention on GPT-6.1 Sol, a generic evaluation cue left success nearly unchanged (63.1% vs 63.4%), the correct objective cue raised it to 91.6%, and the incorrect objective cue cut it to 17.6%.

November 2Monday
  1. Import AI42

    Import AI 479: Spacewar, low-latency tasks versus LLMs, Genie 4

    Import AI 479 covers Realtime-Bench, a 50-task benchmark of 30 web and 20 game tasks built by researchers at UT Austin, Ohio State, UIUC, NVIDIA, Northwestern and Steadyworks to test whether frontier models can act under real-time pressure. GPT-6 Astra (high) scored the highest wall-clock pass rate at 36%, while paused-clock runs let high-reasoning models reach roughly 70% to 80%. The issue also covers the Defense Intelligence Agency's "Challenges to Security in Space" report and Genie 4.

  2. SemiAnalysis58

    SemiAnalysis releases GDPx benchmark for frontier agentic capabilities across high-value domains

    SemiAnalysis released GDPx, a benchmark covering chip design, cybersecurity, biology, finance, legal, quant trading, RSI, hardware design, materials science, alignment and software engineering, built with 14 specialized RL environment partners. Even the best model scores just 12%, with Opus 5.5 winning quant trading, hardware and chip design, Gemini 4 Carbon leading biology and software engineering, and GPT-6 Astra topping RSI and materials science. Every task runs in a realistic simulated environment, such as a ~4,000-file private equity deal room or live multi-service cyber ranges, unlike OpenAI's GDPval, which SemiAnalysis argues spoon-feeds context and lacks environments.

October 30Friday
  1. Microsoft Research40

    Microsoft builds synthetic enterprise data platform to evaluate Copilot

    Microsoft's Industry Evaluation Data Generation Platform creates full-scale synthetic enterprise environments to evaluate Microsoft Copilot without using customer data. Grounded in public sources such as O*NET occupational data, job postings, and SEC filings, it generates realistic tenants with users, meetings, emails, and documents, plus automated query pipelines and quality frameworks. Microsoft says this establishes the first standardized evaluation baseline for Copilot.

October 28Wednesday
  2. Artificial Analysis Articles82

    Gemini 4 Carbon tops Artificial Analysis Intelligence Index v5, ranks second on Coding Agent Index

    Google's Gemini 4 Carbon is the new No. 5 on the Artificial Analysis Coding Agent Index. 5 (max, 62), and it leads 4 of the 9 Index evaluations including HLE-Diamond and Terminal-Bench-Science. 1 rises from 4% to 11%; the model is being rolled out to selected users and is not yet publicly available.

    Why it matters: Gemini 4 Carbon tops Artificial Analysis Intelligence Index v5 at 59 and trails Claude Opus 5.5 by 3 points on the Coding Agent Index, with per-task cost and hallucination figures for comparison.

October 27Tuesday
  1. Ai2 · Allen Institute for AI58

    Ai2 study tests whether LLMs can reliably judge research idea novelty

    Ai2 released Old Ideas, Novel Problems, a study and benchmark testing how reliably LLMs evaluate the novelty of research ideas, along with data, code and a pipeline for building new evaluation sets. Across six judges including GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6, small prompt changes shifted accuracy by more than 50 percentage points on AI-generated ideas, and a simple prompted Claude Opus 4.6 baseline scored 0.854 macro-F1 at $2.59 versus 0.642 at $30.66 for Idea Novelty Checker and 0.772 at $82.05 for Sakana AI's AI Scientist.

  2. Epoch AI · Gradient Updates58

    Epoch AI tests AI on its own research tasks, top models score 65% and fail all

    Epoch AI expanded its earlier effort to automate a senior researcher's job into the Epoch Automation Reports, evaluating AI on real Epoch tasks like graphic design and research planning against the same standards used for human employees. 1 and GPT-6 Astra averaged 65% on the rubric, but a 65% score still counts as a failed task, since every submission would be rejected or revised if a human had made it.

October 26Monday
  1. Import AI62

    Import AI 478: Math for alignment, ProgramBench, and Mirendil's RSI signal

    Import AI 478 rounds up four developments: AI safety startup Resolution autoformalized the resolution of singularities in Lean using orchestrator agents, at an estimated $135,000 in token spend over about a week. Claude Opus 5.5 reached 18.5% resolved on ProgramBench, up from 0% for every model when the benchmark launched in May. OpenAI described agents in separate microVMs exchanging synthetic secrets via processor cache timing, and Mirendil reported one researcher pretrained a 100B-500B parameter coding model in 3.5 months for under $10m in compute.

October 24Saturday
  2. Epoch AI · Gradient Updates42

    Epoch Brief: AI Work Automation Test, SpireClimb Benchmark, and Gemini 4 Argon's Frontier Push

    Epoch AI tested six leading models on 11 realistic Epoch work tasks and found none could reliably meet its quality bar, with Claude Fable 5.1 and GPT-6 Astra roughly tied for the lead. Its new SpireClimb benchmark shows frontier models went from struggling at Slay the Spire to winning over 90% of base-difficulty runs in six months, with GPT-6 Astra at 98%. Gemini 4 Argon scored 166 on the Epoch Capabilities Index, tying Claude Fable 5.1 and trailing Claude Opus 5.5 and GPT-6 Astra at 169.

October 21Wednesday
  1. Amazon Science38

    Amazon Taps AVERI for Independent Review of Its AI Safety Evaluation Datasets

    Amazon partnered with independent nonprofit AVERI to review the quality of its internal safety evaluation datasets, not just model outputs. AVERI validated topical coverage, prompt diversity, and low contamination risk, while recommending broader multilingual coverage, more multistep adversarial prompts, and clearer grading criteria. Amazon responded with refreshed test sets, tiered difficulty levels, and a quarterly review cadence.

  2. OpenAI Alignment Blog28

    OpenAI proposes pair-matched and post-control metrics to extend g-mean² monitorability

    OpenAI's Alignment Blog adds two metrics to g-mean², its dataset-level measure of monitor discrimination. A pair-matched metric compares models only on task instances that support the relevant positive, negative or intervention-valid comparison, while a post-control integration metric simulates monitor-gated rejection and estimates the probability that the deployed system still emits a bad output. The post also derives a pessimistic lower bound for intervention evaluations.

October 20Tuesday
  1. One Useful Thing40

    Ethan Mollick's Co-Existence Charts 30 Months of AI Progress, Made by Claude Opus 5.5

    Ethan Mollick published Co-Existence, his second AI book, 30 months after Co-Intelligence, and used Claude Opus 5.5 in Claude Projects to generate the post's charts and an "otter on an airplane using wifi" video in minutes. He notes 180 models now outperform GPT-4 at release, 19 of which run on a modern smartphone, while METR's Long Horizon Benchmark rose from a four-minute task in 2024 to 17 hours of human work by April 2026 before the benchmark ran out.

  2. AI as Normal Technology40

    Princeton Paper Maps 33 Emerging Trends in AI Agent Evaluation

    A Princeton-led group, with Stanford, the UK AI Security Institute and Brookings, published "Emerging Trends in AI Agent Evaluation," synthesizing 33 trends across evaluation science, research agenda and the evaluator ecosystem. The paper argues agent evaluation is far more complex than model evaluation, since evaluators must scrutinize entire trajectories rather than final outcomes, and notes some agents on the ExploitBench benchmark cost tens of thousands of dollars in credits to evaluate. It calls for an expanded research agenda spanning capabilities, human interaction and institutional impact, plus broader participation and funding for independent evaluators.

October 19Monday
  1. inclusionAI Repositories58

    inclusionAI releases Ming-3.0-tts speech synthesis models in 0.5B dense and 7.9B MoE tiny variants

    inclusionAI released Ming-3.0-tts, an open-weight multilingual speech synthesis family with two models: Ming-3.0-tts-0.5b, a 0.5B dense model, and Ming-3.0-tts-tiny, a MoE model built on Ling V3 with 7.9B total and 1.3B active parameters. Both ship with inference code, a task cookbook and benchmark tables, and support voice cloning, text-based voice design, free-form natural-language delivery control, 14 inline paralinguistic tags, emotion and speed/pitch/volume control, plus Chinese, English, German, Spanish, French, Korean, Japanese, Sichuanese and Cantonese. On the October 2026 Artificial Analysis text-to-speech leaderboards, Ming-3.0-tts ranks 1st among open-weights models on Controlled Voice, 1st overall on Pronunciation Robustness at 93.2%, and 3rd among open-weights models on Provider Voice.

  2. Modal Blog38

    How Legora runs 2,000 parallel agent evals on Modal Sandboxes

    Legora now runs each of its legal agent evals inside a separate Modal Sandbox, booting its full application pinned to a specific commit so every run reflects the exact product version. A typical experiment is 100 cases, 4 harness variants, and 5 trials each — 2,000 runs in parallel — with Modal Volumes snapshotting document, database, and permission state. Legora also moved from gVisor to Modal VM Sandboxes, removing its Docker networking and FUSE workarounds, and cut agent cost by 30% via smarter compaction and custom tool search.

October 12Monday
  1. Import AI38

    Import AI 476: Creativity and RSI, Worse-Than-China Open Weight Models, and AI in Space

    Import AI 476 covers a new benchmark showing frontier LLMs still trail humans on intuitive visual reasoning, with GPT 6 Astra scoring 53.6 pass@1 versus a 93.1 human baseline. It also discusses AI systems improving at "research taste" via the TasteVal benchmark, and argues AI could accelerate space industrialization and create compounding first-mover advantages.

October 10Saturday
  1. Epoch AI · Gradient Updates65

    Epoch AI's InnovationEval finds frontier agents reach only 15% of SDPO's gains on automated AI R&D

    6 Sol and Fable 5 with independently rediscovering a novel post-training technique comparable to Self-Distillation Policy Optimization (SDPO), giving each 3,000 GPU-hours in a sandbox without internet access. 6 Sol's method, largely copied from prior work, reached 15% of SDPO's gains after adjusting for slower training, while Fable 5's technique failed to improve performance.

October 9Friday
  1. Epoch AI · Gradient Updates22

    Epoch Brief, October 8, 2026: AI Agent Population Estimates, Falling AI Costs, and China Chip Exposure

    Epoch AI estimates that AI chips shipped through 2027 could support about 30 to 170 million concurrent frontier-model agents, or almost 2 billion if run on cheaper models. Its research also finds the cost of a fixed level of AI performance has fallen about 47% per quarter over three years, roughly 13× per year. Separately, Epoch estimates China's exposure to semiconductor supply shocks is 2.7× that of the US.

October 6Tuesday
  1. LiveKit Blog38

    Retell vs. Vapi vs. LiveKit: A Developer Comparison for Voice AI Agents

    LiveKit published a developer-focused comparison of Retell, Vapi and LiveKit for building voice AI agents, framing the choice around four questions: who iterates on the agent after launch, what happens when needs fall outside defaults, whether it must run beyond phone calls, and cost at scale. Retell is a managed platform with a visual editor, Vapi a developer-oriented managed platform configured via API and dashboard, and LiveKit an Apache 2.0 open source framework with self-hosting. Estimated all-in cost per minute runs about $0.070–$0.310 for Retell, $0.070–$0.116 for Vapi and $0.037–$0.099 for LiveKit.

October 5Monday
  1. GitHub Blog · AI & ML58

    GitHub releases ReviewBench, an open benchmark for AI code review

    GitHub released ReviewBench, an open benchmark for AI code review agents built on 219 public pull requests across 19 languages, with language and repository-size distributions modeled on 103.9 million GitHub pull requests. The golden set draws findings from human reviewers, author follow-up commits, static analysis and multiple frontier LLMs, graded by Claude Sonnet 5 under a published rubric, and senior engineers independently re-labeled every finding with 96.6% agreement. GitHub says its offline evaluation of Copilot code review anticipated the direction of a production A/B test, where addressed rate rose 8.0%, recall rose 13.6% and cost per review fell 8.0%.

September 28Monday
  1. Anthropic · Claude.dev72

    Anthropic adds build-eval and hillclimb commands to the claude-api skill in Claude Code

    Anthropic added two commands to the claude-api skill inside Claude Code: /claude-api build-eval builds an evaluation set in your codebase, and /claude-api hillclimb iterates on an application against that evaluation with a held-out set to catch overfitting.

    Why it matters: Anthropic adds build-eval and hillclimb commands to its claude-api skill, showing how Claude Code can design evaluations and iterate against them while guarding against overfitting.

September 22Tuesday
  1. Every75

    Vibe Check: GPT-6 Sol vs. Claude Opus 5.5

    Every published a Vibe Check comparing OpenAI's GPT-6 Sol and Anthropic's Claude Opus 5.5, both released the same day, to help readers decide which model to use. The comparison draws on the outlet's own experience and evaluations of both models on its daily work, and the full article is available to subscribers.

September 21Monday
  1. Every44

    How to Create Your Own Personal AI Benchmark

    Every's head of evals explains how to build a personal AI benchmark instead of relying on MMLU-Pro and other public tests, drawing on his own experience testing Claude Opus 5, Claude Fable 5, GPT-5.6 Sol, Astra, Luna and Haiku on real work tasks. The approach has three levels and is meant to show when to switch models and whether a smaller, cheaper one suffices.

September 18Friday
  1. Every42

    Vibe Check: Is GPT-6 Astra a Breakthrough for Indie Filmmakers?

    Creative studio Afterimage tested GPT-6 Astra on footage from real shoots to see whether its Blender 3D skills carried over to visual effects work. Astra handled tasks ranging from replacing a dish in a moving pan to adding a building to a Brooklyn street and making a giant parade float appear outside a window in handheld iPhone footage. Some shots worked and others didn't, but the team ended up dreaming up effects they'd never have attempted before.

August 24Monday
August 18Tuesday
  1. Vapi Blog55

    Vapi launches Simulations to test voice agents against AI callers

    Vapi launched Simulations, a native AI-powered testing feature that places AI testers with distinct personalities and customer-derived scenarios on the other end of a call with a voice agent. Each run returns pass/fail results, a full transcript and a recording, and teams can mock API responses, run cheap text-based chat mode or full voice mode, and trigger runs via API with quality gates that block deploys when pass rates drop.

August 5Wednesday
August 3Monday
  1. Vapi Blog33

    Why model flexibility matters in voice: Speechify Simba 3.2 and Fish Audio S2.1-Pro top Vapi's Humanness Index

    Speechify's Simba 3.2 debuted atop Vapi's Humanness Index at 99, one point under the human baseline of 100, and Fish Audio's S2.1-Pro entered at 94 with 141 ms time to first audio. The prior leader, xAI's Grok TTS, held the top spot for only three weeks at 95, and the leading score has since moved from 95 to 99. Vapi argues that model-agnostic voice platforms let teams swap TTS models as rankings shift.