Skip to content
Only Research#Benchmarks
November 13Friday
  1. Hacker News · AI58

    How much of GitHub's most-starred AI tooling is written by coding agents?

    An analysis of the full git history of 52 of GitHub's most-starred AI-tooling repositories, covering 324,088 commits, finds that in the median repo 16.9% of commits in its last 180 days of activity carry an AI signature, and 21 of 52 sit at 25% or more. Stars and AI-signed share show no detectable link (Spearman ρ = -0.05, p = 0.72), and 85.3% of the 36,892 signed commits come from Claude, whose tool adds a trailer by default. The author cautions that heavy AI use is concentrated in projects started in 2025 or later, so the sample cannot separate AI use from recency, and every AI percentage is a lower bound.

  2. Hacker News · AI38

    AI Models Know WPS Principles but Fail to Apply Them, Benchmark Finds

    Our Secure Future's WPS AI Leaderboard benchmark, built on the Collective Intelligence Project's Weval platform, finds LLMs systematically omit gender dimensions from peace and security analysis even though they can recite UNSCR 1325. The WPS Competence Gap appears as passive omission, where models drop women and girls from routine assessments, and active capitulation, where they help decision-makers route around policy.

  4. Hacker News · AI58

    Westphalia Labs tests 11 AI models chairing a fictional seven-government negotiation

    Westphalia Labs, an evaluation group founded by a former diplomat, ran eleven AI models as chair of a fictional seven-government negotiation, forty runs each across a possible and an impossible version. Only GPT-6.1 Sol passed every run, reaching agreement when possible and declaring consensus impossible when it was; Claude Opus 5.5, Grok 4.7 and the low-cost DeepSeek V4.1 Flash each called the impossible version correctly every time and agreed in 18 of 20 possible runs. Overreach caused 39 failed runs against 13 from conceding too little, and a likely forecast was followed by agreement in 72 of 110 cases.

  5. The Verge · AI58

    Study finds ChatGPT, Gemini and Perplexity recommend fake retail websites

    A study by IP protection firm Podqi found that ChatGPT, Gemini and Perplexity all cited or displayed risky retail websites, defined as fake storefronts, reregistered old brand domains or dead sites, in answers to shopping questions about 151 brands. Across 6,795 answers collected in two September testing periods, Perplexity surfaced risky sites in 15 percent of responses, ChatGPT in 8 percent and Gemini in 4 percent, while ChatGPT warned about the risky site it surfaced 24 percent of the time. Queries about smaller brands produced risky sites about twice as often as those about larger companies, and prompts about regional brand websites triggered them more frequently, with Perplexity once naming the wrong page as Stüssy's official site.

November 12Thursday
  1. Epoch AI · Gradient Updates64

    Epoch Brief: GLM 5.3 autonomously builds Firefox exploits from patch gaps for about $1,000 each

    Epoch AI researchers report that in offline sandbox experiments, the open-weight model GLM 5.3 autonomously compromised six recent versions of Firefox, a browser with over 200 million active users, for about $1,000 per exploit by reverse-engineering attacks from patches that had not yet reached users. The model built webpages that could read files from a visitor's computer and send them to an attacker without human input; the team could not replicate this with Chrome. The same brief notes that Anthropic's Claude Fable 5.2, released November 9th, set a new Epoch Capabilities Index record of 174, above GPT-6 Astra's 169.

November 11Wednesday
  1. inclusionAI Models40

    inclusionAI Releases ZoomGain-9B and ZoomGain-4B Models with ZoomGainBench and RL Training Data

    inclusionAI released the ZoomGain-9B and ZoomGain-4B vision-language model checkpoints, alongside the ZoomGainBench evaluation benchmark and ZoomGain RL training data. ZoomGain mines global-to-zoom perception gaps to build fine-grained visual supervision, and ZoomGain-9B reached superior performance on multiple fine-grained and general perception benchmarks. ZoomGainBench contains 1,000 questions over 941 images, while the RL dataset holds 9,000 training records and 7,713 source images.

November 10Tuesday
  1. Apple Machine Learning Research31

    Apple study evaluates LLM judges for personalized writing style assessment

    Apple researchers tested how well LLM judges assess whether generated text matches a user's preferred writing style, generating text with four LLMs across three long-form writing tasks and comparing judge annotations against human pairwise labels. Judge quality correlated only weakly with general LLM ability on MMLU (Pearson's ρ ≤ 0.2) and varied by writing task and evaluation criteria, and paired comparisons proved more consistent than independent scoring. The strongest LLMs did not most reliably reproduce human evaluations, possibly due to exaggerated sensitivity to subtle details.

  2. Artificial Analysis Articles58

    Google DeepMind and Artificial Analysis launch MKA benchmark for agentic knowledge work on complex visual files

    Google DeepMind partnered with Artificial Analysis to launch the Multimodal Knowledge Agent (MKA) benchmark, which tasks agents with reviewing dense visual project files such as architectural drawings, engineering schematics and clinical scans to produce slide decks, spreadsheets, PDFs, Word documents and marked-up images.

November 9Monday
  1. Artificial Analysis Articles58

    Artificial Analysis releases Endpoint Accuracy Index v1.1 with tighter long-context intervals and six-model coverage

    Artificial Analysis launched Endpoint Accuracy Index v1.1, which tightens AA-LCR-25 confidence intervals and adds coverage for GLM-5.3 Flash and Muse Spark Open. Across 103 serverless endpoints benchmarked, 24 score statistically below the self-hosted reference, with causes including output token caps, reduced reasoning, tool call formatting errors and request timeouts. Raising AA-LCR-25 to 25 repeats per question narrows its average 95% confidence interval from ±4.7 to ±2.6 points, and the composite index interval from ±5.5 to ±4.3 points across the 45 endpoints scored under both versions.

November 8Sunday
  1. Artificial Analysis Articles38

    Artificial Analysis Decision Index v2 Released with New Decision Score and Private Benchmarks

    Artificial Analysis released Decision Index v2, with 45% of the Index now built on private datasets, up from 25% in v1. Drex Max v1.1 from Nace AI narrowly leads decision models at 73, within 1 point of GLiDE v1.2 and Mammoth 27B. The new Decision Score rewards both correctness and calibrated confidence, while AA-Paperclip and AA-Symmetry add private document and consistency tests.

November 6Friday
  1. SemiAnalysis75

    SemiAnalysis benchmarks Google TPUv7 Ironwood on agentic inference against NVIDIA GPUs

    SemiAnalysis released the first long-context, multi-turn agentic inference results on InferenceX/AgentX for Google TPUv7 Ironwood across four models, finding it competitive on performance per dollar or state-of-the-art in several cases. On GLM-5.2 in FP8, TPUv7 delivered 1.2x to 2.7x the tokens per dollar of B200, B300 and MI355X from 60 to 300 tok/s/user, while FP4 GPUs still led on GLM-5.3 because TPUv7 lacks native FP4 support until TPUv8 in Q1 2027. The piece also argues NVIDIA's bureaucracy slows driver and ecosystem contributions compared with AMD and Google.

  2. Artificial Analysis Articles40

    VIALS-AA: Different Paths to Scientific Image Reasoning

    VIALS-AA analyzes how 19 models reason over scientific images, classifying 9,177 rollouts of tool calls. Viewing images accounted for 38.3% of calls and image preparation or measurement 47.3%, while image measurement was just 5–6% of calls for GPT-6 Astra (max) and GPT-6.1 Sol (max) versus 25–27% for Claude Opus 5.5 (max) and Claude Sonnet 5.5 (max). Gemini 3.8 Flash (high) produced 86.7% of the cohort's commentary/no-op calls, and cost per task varies by nearly 100× across models.

November 3Tuesday
  1. Ai2 · Allen Institute for AI55

    Ai2 releases StateBench to test how language models track state changes across long documents

    Ai2 introduced StateBench, a benchmark with four difficulty levels for long-context state tracking, from immutable assignment to self-referential and cross-referential updates. In tests, Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna handled the easier levels but dropped to 13% or lower on the longest general finite problems, while 275M-parameter Gated DeltaNet and hybrid models trained from scratch generalized almost perfectly on the first three levels and transformers never learned the hardest one.

  2. OpenAI Alignment Blog58

    OpenAI examines eval-production behavior gaps in agent safety and the role of eval-awareness

    OpenAI researchers measured an eval-production behavior gap in agent safety: across six served models, success on the Respecting Warnings evaluation was 10.2 to 37.4 percentage points lower than on matched Codex production decisions, with the largest gaps in GPT-5.6 Sol and GPT-6 Sol. General eval-awareness alone showed weak association with the gap (R² = 0.008), while specific eval-objective awareness was stronger (R² = 0.370), and models most often verbalizing the wrong objective (resisting prompt injection) had the biggest gaps. In an intervention on GPT-6.1 Sol, a generic evaluation cue left success nearly unchanged (63.1% vs 63.4%), the correct objective cue raised it to 91.6%, and the incorrect objective cue cut it to 17.6%.

November 2Monday
  1. Import AI42

    Import AI 479: Spacewar, low-latency tasks versus LLMs, Genie 4

    Import AI 479 covers Realtime-Bench, a 50-task benchmark of 30 web and 20 game tasks built by researchers at UT Austin, Ohio State, UIUC, NVIDIA, Northwestern and Steadyworks to test whether frontier models can act under real-time pressure. GPT-6 Astra (high) scored the highest wall-clock pass rate at 36%, while paused-clock runs let high-reasoning models reach roughly 70% to 80%. The issue also covers the Defense Intelligence Agency's "Challenges to Security in Space" report and Genie 4.

  2. SemiAnalysis58

    SemiAnalysis releases GDPx benchmark for frontier agentic capabilities across high-value domains

    SemiAnalysis released GDPx, a benchmark covering chip design, cybersecurity, biology, finance, legal, quant trading, RSI, hardware design, materials science, alignment and software engineering, built with 14 specialized RL environment partners. Even the best model scores just 12%, with Opus 5.5 winning quant trading, hardware and chip design, Gemini 4 Carbon leading biology and software engineering, and GPT-6 Astra topping RSI and materials science. Every task runs in a realistic simulated environment, such as a ~4,000-file private equity deal room or live multi-service cyber ranges, unlike OpenAI's GDPval, which SemiAnalysis argues spoon-feeds context and lacks environments.

October 30Friday
  1. Microsoft Research40

    Microsoft builds synthetic enterprise data platform to evaluate Copilot

    Microsoft's Industry Evaluation Data Generation Platform creates full-scale synthetic enterprise environments to evaluate Microsoft Copilot without using customer data. Grounded in public sources such as O*NET occupational data, job postings, and SEC filings, it generates realistic tenants with users, meetings, emails, and documents, plus automated query pipelines and quality frameworks. Microsoft says this establishes the first standardized evaluation baseline for Copilot.

October 27Tuesday
  1. Ai2 · Allen Institute for AI58

    Ai2 study tests whether LLMs can reliably judge research idea novelty

    Ai2 released Old Ideas, Novel Problems, a study and benchmark testing how reliably LLMs evaluate the novelty of research ideas, along with data, code and a pipeline for building new evaluation sets. Across six judges including GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6, small prompt changes shifted accuracy by more than 50 percentage points on AI-generated ideas, and a simple prompted Claude Opus 4.6 baseline scored 0.854 macro-F1 at $2.59 versus 0.642 at $30.66 for Idea Novelty Checker and 0.772 at $82.05 for Sakana AI's AI Scientist.

  2. Epoch AI · Gradient Updates58

    Epoch AI tests AI on its own research tasks, top models score 65% and fail all

    Epoch AI expanded its earlier effort to automate a senior researcher's job into the Epoch Automation Reports, evaluating AI on real Epoch tasks like graphic design and research planning against the same standards used for human employees. 1 and GPT-6 Astra averaged 65% on the rubric, but a 65% score still counts as a failed task, since every submission would be rejected or revised if a human had made it.

October 24Saturday
  2. Epoch AI · Gradient Updates42

    Epoch Brief: AI Work Automation Test, SpireClimb Benchmark, and Gemini 4 Argon's Frontier Push

    Epoch AI tested six leading models on 11 realistic Epoch work tasks and found none could reliably meet its quality bar, with Claude Fable 5.1 and GPT-6 Astra roughly tied for the lead. Its new SpireClimb benchmark shows frontier models went from struggling at Slay the Spire to winning over 90% of base-difficulty runs in six months, with GPT-6 Astra at 98%. Gemini 4 Argon scored 166 on the Epoch Capabilities Index, tying Claude Fable 5.1 and trailing Claude Opus 5.5 and GPT-6 Astra at 169.

October 21Wednesday
  1. OpenAI Alignment Blog28

    OpenAI proposes pair-matched and post-control metrics to extend g-mean² monitorability

    OpenAI's Alignment Blog adds two metrics to g-mean², its dataset-level measure of monitor discrimination. A pair-matched metric compares models only on task instances that support the relevant positive, negative or intervention-valid comparison, while a post-control integration metric simulates monitor-gated rejection and estimates the probability that the deployed system still emits a bad output. The post also derives a pessimistic lower bound for intervention evaluations.

October 20Tuesday
  1. AI as Normal Technology40

    Princeton Paper Maps 33 Emerging Trends in AI Agent Evaluation

    A Princeton-led group, with Stanford, the UK AI Security Institute and Brookings, published "Emerging Trends in AI Agent Evaluation," synthesizing 33 trends across evaluation science, research agenda and the evaluator ecosystem. The paper argues agent evaluation is far more complex than model evaluation, since evaluators must scrutinize entire trajectories rather than final outcomes, and notes some agents on the ExploitBench benchmark cost tens of thousands of dollars in credits to evaluate. It calls for an expanded research agenda spanning capabilities, human interaction and institutional impact, plus broader participation and funding for independent evaluators.

October 12Monday
  1. Import AI38

    Import AI 476: Creativity and RSI, Worse-Than-China Open Weight Models, and AI in Space

    Import AI 476 covers a new benchmark showing frontier LLMs still trail humans on intuitive visual reasoning, with GPT 6 Astra scoring 53.6 pass@1 versus a 93.1 human baseline. It also discusses AI systems improving at "research taste" via the TasteVal benchmark, and argues AI could accelerate space industrialization and create compounding first-mover advantages.

October 10Saturday
  1. Epoch AI · Gradient Updates65

    Epoch AI's InnovationEval finds frontier agents reach only 15% of SDPO's gains on automated AI R&D

    6 Sol and Fable 5 with independently rediscovering a novel post-training technique comparable to Self-Distillation Policy Optimization (SDPO), giving each 3,000 GPU-hours in a sandbox without internet access. 6 Sol's method, largely copied from prior work, reached 15% of SDPO's gains after adjusting for slower training, while Fable 5's technique failed to improve performance.

October 5Monday
  1. GitHub Blog · AI & ML58

    GitHub releases ReviewBench, an open benchmark for AI code review

    GitHub released ReviewBench, an open benchmark for AI code review agents built on 219 public pull requests across 19 languages, with language and repository-size distributions modeled on 103.9 million GitHub pull requests. The golden set draws findings from human reviewers, author follow-up commits, static analysis and multiple frontier LLMs, graded by Claude Sonnet 5 under a published rubric, and senior engineers independently re-labeled every finding with 96.6% agreement. GitHub says its offline evaluation of Copilot code review anticipated the direction of a production A/B test, where addressed rate rose 8.0%, recall rose 13.6% and cost per review fell 8.0%.

August 5Wednesday
April 29Wednesday
  1. Factory58

    Factory benchmarks 13 models on code review, finds GPT-5.2 best and MiniMax M2.7 cheapest

    Factory built an open-source benchmark testing 13 models on 50 real pull requests from Sentry, Grafana, Keycloak, Discourse and Cal.com, with each model reviewing every PR at least three times at high reasoning effort and an LLM judge scoring findings against a human-curated golden set. GPT-5.2 led at 60.5% F1 for $1.25 per PR, ahead of Claude Opus 4.6 at 59.8% F1 and $3.11 per PR, while MiniMax M2.7 reached 45.6% F1 at $0.15 per PR and Kimi K2.5 hit 51.9% F1 at $0.41. Factory says cost explains only about 21% of quality variance, and it open-sourced the golden set, evaluation scripts and a Droid Action GitHub Action supporting all 13 models.