#评测/基准

NukeTorch: Four MacBook Pro M5 Max Macs Serve NVIDIA GR00T 2.45x Faster Than Fastest Published H100 Result

Fenon AI's NukeTorch engine ran NVIDIA's GR00T N1.7 robot policy on four MacBook Pro M5 Max machines at 152.07 action chunks per second with 192 clients, 2.45x the fastest published H100 figure and 4.24x NVIDIA's own published number. For a single request the H100 still wins at 16.08 ms versus 44.34 ms, and the four Macs list at $19,996 against at least $25,000 for one H100 card. The one-Mac version is a free download, while the multi-machine build requires an evaluation license.

Gemini 4 Hydrogen (High) Intelligence, Performance and Price Analysis

An analysis of Gemini 4 Hydrogen (High) compares its intelligence, performance and price, using capability indexes that measure model performance on specific capabilities and industries. The comparison covers token use, cost and context window, alongside performance metrics including time per task in minutes, output speed in tokens per second, and time to first answer token and first token in seconds.

Google Releases Gemini 4 Hydrogen, Claiming 2.5x Speed Over Gemini 3.8 Flash

Google released Gemini 4 Hydrogen, a closed-weights model from Google DeepMind positioned below Gemini 4 Carbon and Argon as a daily driver for coding and agentic workflows. It offers a 1,048,576-token unified window, four thinking levels with high as the default, and API pricing of $1.50/$7.50 per 1M input/output tokens, flat across the full window. Google reports 57% on Terminal-Bench 4.1 versus 18.2% for Gemini 3.8 Flash and 51 on the Artificial Analysis Intelligence Index, but the model ranks only #5 on Agent Arena; pairing it with Carbon in Antigravity used 24% less quota and finished 19% faster than Carbon alone.

Same event; the pick carries“Google DeepMind releases Gemini 4 Hydrogen, a faster Gemini 4 model for coding and agentic workflows”

Stanford CS 329Z: Engineering AI Agents Course Covers Agent Design, RAG, Tools and Multi-Agent Systems

Stanford's CS 329Z, Engineering AI Agents, walks through the anatomy of agentic systems — LLM core, reasoning, memory, tools and environment — and the gap between compound AI systems and true agents. The course covers RAG pipelines, tool use and function calling with MCP, DSPy and LangGraph orchestration, memory architectures such as MemGPT, and multi-agent orchestration patterns. It closes with optimization methods including GEPA, PPO and GRPO, plus a guest lecture from the Terminal-Bench creators.

Sovereign: a router that learns which model to trust without labels

Solibra built Sovereign, a router that sits between an application and model providers and learns per-request model choice from grades of its own finished work, with no answer key and no prior knowledge of the models. Across eight public benchmarks run against Claude Opus 5.5 and Claude Haiku 5.5, Sovereign spent 43% to 81% less than Opus on every benchmark, matched or beat Opus accuracy on USACO, LiveCodeBench and SATBench, and was more accurate than Haiku everywhere, by up to 55 points on USACO. On USACO its accuracy rose from 88% over the first 25 tasks to 97.7% by the end, passing Opus around task 150, and its grader agreed with the hidden answer key 95% of the time.

OpenRouter tests 11 rerankers on a 274-question RAG benchmark

OpenRouter benchmarked 11 rerankers on a 274-question synthetic RAG dataset and found that reranking the top 30 vector-search candidates with voyageai/rerank-3-lite raised precision@1 from 81.0% to 97.1% at about $0.10 per 1,000 queries. The guide recommends retrieving roughly 30 chunks, reranking them, and sending only the top 3 to the LLM, which in its test lifted answer accuracy to 99.3% while cutting answering cost from $0.22 to $0.16 per 1,000 questions. It also flags failure modes: omitting the input_type parameter cost 7 to 12 points of precision@1, and cohere/rerank-v3.5 silently ignores content beyond its 4,096-token limit.

How much of GitHub's most-starred AI tooling is written by coding agents?

An analysis of the full git history of 52 of GitHub's most-starred AI-tooling repositories, covering 324,088 commits, finds that in the median repo 16.9% of commits in its last 180 days of activity carry an AI signature, and 21 of 52 sit at 25% or more. Stars and AI-signed share show no detectable link (Spearman ρ = -0.05, p = 0.72), and 85.3% of the 36,892 signed commits come from Claude, whose tool adds a trailer by default. The author cautions that heavy AI use is concentrated in projects started in 2025 or later, so the sample cannot separate AI use from recency, and every AI percentage is a lower bound.

AI Models Know WPS Principles but Fail to Apply Them, Benchmark Finds

Our Secure Future's WPS AI Leaderboard benchmark, built on the Collective Intelligence Project's Weval platform, finds LLMs systematically omit gender dimensions from peace and security analysis even though they can recite UNSCR 1325. The WPS Competence Gap appears as passive omission, where models drop women and girls from routine assessments, and active capitulation, where they help decision-makers route around policy.

Gemini 4 Hydrogen (Xhigh) Intelligence, Performance and Price Analysis

An analysis of Gemini 4 Hydrogen (Xhigh) covers its intelligence and capability indexes across specific capabilities and industries, along with token use cost and context window. Performance is measured by time per task in minutes, output speed in tokens per second, and time to first answer token and first token in seconds.

AI search is recommending fake retail websites

A study drawing on 6,795 answers from the logged-out consumer versions of ChatGPT, Gemini and Perplexity finds AI search recommending fake retail websites. The research was run with Gauge, an AI search company that helps brands appear in more AI answers.

Terminal-Bench 4.1 Released with Hardened Verifiers Against Reward Hacking

Terminal-Bench 4.1 is released with improved verifiers designed to resist reward hacking, and regrading TB 4.0 with the new verifiers required $0 in additional inference cost. Only about 1.16% of 11,550 trials changed score, and all 35 agents' updated scores remain within their original TB 4.0 95% confidence intervals.

Westphalia Labs tests 11 AI models chairing a fictional seven-government negotiation

Westphalia Labs, an evaluation group founded by a former diplomat, ran eleven AI models as chair of a fictional seven-government negotiation, forty runs each across a possible and an impossible version. Only GPT-6.1 Sol passed every run, reaching agreement when possible and declaring consensus impossible when it was; Claude Opus 5.5, Grok 4.7 and the low-cost DeepSeek V4.1 Flash each called the impossible version correctly every time and agreed in 18 of 20 possible runs. Overreach caused 39 failed runs against 13 from conceding too little, and a likely forecast was followed by agreement in 72 of 110 cases.

The Singularity Clock v1.4.0 Tracks AGI, Singularity and ASI Timelines

The Singularity Clock v1.4.0 is a public measurement instrument that projects AGI around April 2027, the Singularity around December 2029 and ASI around December 2031, all holding flat since November. Its 14 signals include METR task horizon at 11.4 hours (+38% QoQ), SWE-bench Verified at 88.2%, Humanity's Last Exam at 38.9% and ARC-AGI-2 at 62.4%. The project reports a capability-to-safety velocity ratio of 1.77× and publishes its methodology and code on GitHub under CC-BY-4.0.

Have AI timelines suddenly moved? Re-examining Anthropic's acceleration and the flat frontier

A review of METR's report on AI R&D acceleration at Anthropic and the follow-on analyses finds Anthropic's capabilities growth shifted trend around October 2025, near Opus 4.5, but competing labs did not speed up and the overall frontier has stayed on its prior trend. The author weighs compute growth, which rose about 5.7-fold at Anthropic in 2025 versus 4.5-fold at OpenAI and 3.8-fold at Google DeepMind, against AI building AI, where METR attributes roughly 80% of the acceleration using redacted data outsiders cannot verify. Depending on which scenario holds, the chance of a superhuman AI researcher in 2027 ranges from about 70% under METR's headline figure to about 20% using only public Epoch numbers and the field's earlier trajectory.

Google releases Gemini 4 Hydrogen, scoring above the larger Argon four weeks after Argon's launch

Google has released Gemini 4 Hydrogen, a fast general-purpose model for developers and everyday users, priced at $1.50 per million input tokens and $7.50 per million output tokens with a one-million-token context window. Google says Hydrogen scores 51 on the Artificial Analysis Intelligence Index at its default high thinking level and 54 at the new xhigh level, versus 50 for the larger Gemini 4 Argon. Independent testing by Artificial Analysis puts Hydrogen roughly on par with Claude Sonnet 5.5 at maximum thinking, but weaker on agentic terminal work, scoring 54 percent on Terminal-Bench 4.0 against 64 percent for Claude Sonnet 5.5.

Same event; the pick carries“Google DeepMind releases Gemini 4 Hydrogen, a faster Gemini 4 model for coding and agentic workflows”

What Is Eval-Driven Development? How to Improve AI Agents

LangChain explains eval-driven development (EDD), a three-step process for building AI agents: define what good looks like, write the eval before you build, and compare results after each change. Because agents are non-deterministic, evals score runs across a dataset of examples and can use LLM-as-a-judge, with real agent runs in LangSmith as a source of test inputs.

Study finds ChatGPT, Gemini and Perplexity recommend fake retail websites

A study by IP protection firm Podqi found that ChatGPT, Gemini and Perplexity all cited or displayed risky retail websites, defined as fake storefronts, reregistered old brand domains or dead sites, in answers to shopping questions about 151 brands. Across 6,795 answers collected in two September testing periods, Perplexity surfaced risky sites in 15 percent of responses, ChatGPT in 8 percent and Gemini in 4 percent, while ChatGPT warned about the risky site it surfaced 24 percent of the time. Queries about smaller brands produced risky sites about twice as often as those about larger companies, and prompts about regional brand websites triggered them more frequently, with Perplexity once naming the wrong page as Stüssy's official site.

Amazon Bedrock AgentCore Evaluations: Built-in, DeepEval and AutoEval evaluators

Amazon Bedrock AgentCore Evaluations now lets teams run AWS built-in evaluators alongside managed DeepEval and AutoEval metrics, plus custom evaluators derived from either. The built-in catalog spans 15 evaluators across response quality, goal completion, tool use, skills and safety, with three trajectory checks for expected tool paths. DeepEval adds multi-turn metrics such as ConversationCompleteness and KnowledgeRetention, while AutoEval contributes specialized signals including security.

AWS open-sources UAEF, a framework-agnostic evaluation system for AI agents

AWS has open-sourced UAEF (Universal Agent Evaluation Framework), a framework-agnostic system that uses adapters to normalize agent traces from LangGraph, LangChain, Strands Agents, Amazon Bedrock AgentCore and Langfuse into one canonical AgentTrace, then scores them with 36 built-in metrics across 7 dimensions. Results are stored per experiment in Amazon DynamoDB and Amazon S3, so teams can compare agent versions against a baseline, flag regressions with a default 5 percent threshold, and generate statistical reports plus LLM-written insights. The post says customers across 12 organizations in telecom, healthcare and financial services used UAEF before release, and the Apache 2.0 code, Python library, CDK deployment and per-framework notebooks are in the awslabs/uaef GitHub repository.

Agent DLC: Engineering AI Agents from Prototype to Production

AWS introduces the Agent Development Lifecycle (Agent DLC), a framework with one lifecycle, two loops, and a single explicit standard for judging every agent change. It addresses three agent properties that break conventional CI/CD: non-determinism, prompts as source code, and independently moving dependencies. A reference HR policy assistant shows chunk-level retrieval precision of 1.00 across all three test questions while fact-level precision fell to 0.09 and retrieval coverage hit 0.00 on one question.

How to Benchmark LLMs: A Code-Graded Guide from OpenRouter

OpenRouter published a guide for benchmarking LLMs on your own application's inputs, grading outputs with code and comparing accuracy, billed cost and request time. On its 60 synthetic tasks, GPT-6.1 Sol passed 180/180 trials across three repeats, while DeepSeek had the lowest response cost at 157/180 passes. Agreement with the public ranking was weak, with a Spearman correlation of 0.109.

Google releases Gemini Decision 1 decision model API with multi-modal input

Google has released Gemini Decision 1, a decision model API that supports yes/no, choice and rubric-scoring questions, joining Jev and OpenAI in the decision model category. It is priced at 10 cents per million input tokens with free output, and unlike Jev's text-only input it accepts text, images, audio, video and PDFs. In a test on a robot video, Gemini returned probabilities such as 0.9234 for robot and 0.0498 for dragon instead of the 1.0 or 0.0 that Jev and OpenAI return.

Epoch Brief: GLM 5.3 autonomously builds Firefox exploits from patch gaps for about $1,000 each

Epoch AI researchers report that in offline sandbox experiments, the open-weight model GLM 5.3 autonomously compromised six recent versions of Firefox, a browser with over 200 million active users, for about $1,000 per exploit by reverse-engineering attacks from patches that had not yet reached users. The model built webpages that could read files from a visitor's computer and send them to an attacker without human input; the team could not replicate this with Chrome. The same brief notes that Anthropic's Claude Fable 5.2, released November 9th, set a new Epoch Capabilities Index record of 174, above GPT-6 Astra's 169.

inclusionAI Releases LemonDecider 4B v3 Decision Model Based on Qwen3.5-4B

inclusionAI released LemonDecider 4B v3, a decision model fine-tuned from Qwen3.5-4B that returns choice, yes/no or ordinal probabilities via a single joint forward pass instead of generating answer tokens. It was trained on about 1.20M decision/QA rows and scored 61.67 on the public Decision Index 0.4 suite, with median request latency of 49.9 ms.

OpenRouter tests what fp8 and fp4 quantization do to LLM quality and price

OpenRouter ran 280 exact-answer tasks against six endpoints of Qwen3.8 27B on October 20, 2026 and did not detect an accuracy difference between fp16, fp8 and fp4, with all six scoring between 98.3% and 99.2%. Across 1,455 endpoints, 59.5% report no precision at all and fp8 is the most common labeled format at 23.6%. Lower precision was cheaper for most models but not all, and provider choice moved price more than precision did, with the priciest same-precision endpoint a median 1.72 times the cheapest.

JioStar Chief Architect Vijay Seshadri on AI Discovery at 500M-User Scale

JioStar chief architect Vijay Seshadri shares lessons on running AI discovery at 500M-user scale, covering evals, guardrails, and taking AI into production. The OpenAI-published piece focuses on the operational practices behind large-scale AI deployment.

inclusionAI Releases ZoomGain-9B and ZoomGain-4B Models with ZoomGainBench and RL Training Data

inclusionAI released the ZoomGain-9B and ZoomGain-4B vision-language model checkpoints, alongside the ZoomGainBench evaluation benchmark and ZoomGain RL training data. ZoomGain mines global-to-zoom perception gaps to build fine-grained visual supervision, and ZoomGain-9B reached superior performance on multiple fine-grained and general perception benchmarks. ZoomGainBench contains 1,000 questions over 941 images, while the RL dataset holds 9,000 training records and 7,713 source images.

inclusionAI Releases SingVL S1 4B Multimodal Decision Model on Qwen3.5

inclusionAI released SingVL S1, a 4B multimodal decision model built on Qwen3.5 that answers named questions over text or JSON and up to four images in a single Transformers forward pass. It supports choice, yes/no (noul) and 2–10 level score questions, scoring 73.94 on the Public Vision Index and 69.76 on the Public Text Index. The model ships with Python inference code and a FastAPI HTTP server.

NaiveAI's Naive-N0.5-Flash claims 2,100 tokens per second: how well does it code?

Latent Space tested NaiveAI's open-weights Naive-N0.5-Flash, a coding model derived from Xiaomi's MiMo-V2.5 and paired with the NaiveRT inference engine, against DeepSeek V4.1 Flash 1017 and GLM 5.3 Flash. In the author's runs it peaked near 2,000 tokens per second and sustained about 1,000 during thinking-heavy tasks, with a median response time of 3.1 seconds versus 5.3 and 10.1 seconds for the two comparison models. On hard HLE questions it averaged about 108,500 completion tokens with thinking on, roughly nine times GLM 5.3 Flash and 11 times DeepSeek V4.1 Flash 1017, and the author notes the sample was 10 questions per difficulty level with no accuracy grading.

Meta AI launches Muse Ember, its smallest and most efficient model

Meta AI launched Muse Ember, described as its smallest and most efficient model, aimed at tool calls, coding subtasks, routing, extraction and long document parsing. For shorter prompts it is priced at $0.10 per million input tokens and $0.50 per million output, the same as GPT-6 Luna and Claude Haiku 5.5, while its long-prompt rates of $0.30 input and $1.50 output above 200k tokens place it between the two. Simon Willison notes the model sits on the Pareto line of Artificial Analysis's Intelligence Index vs. Cost per Task chart, and that its six reasoning levels produced pelican SVGs of comparable quality to Muse Wonder at 40x the price.

Anthropic releases Claude Fable 5.2 and Mythos 5.2, retaking the top LLM spot as its IPO roadshow begins

Anthropic released Claude Fable 5.2 and Claude Mythos 5.2, which top most independent leaderboards, including Artificial Analysis Intelligence Index v5 at 65 and SemiAnalysis GDPx at 16.4%. Pricing drops 20% to $8/$40 per 1M input/output tokens, but SemiAnalysis reports cost per GDPx task rose from $67.16 to $88.15 because the model writes hundreds to thousands of test cases and spawns subagents to check its work. Multiple evals show the max effort level underperforming high, and Fable is capped at 50% of a weekly subscription limit.

OpenRouter benchmarks 15 vision models and Mistral OCR on scanned documents

OpenRouter tested 15 vision models, Mistral OCR and Tesseract on eight self-made degraded scans, and Claude Opus 5.5 was most accurate with 262 of 264 key values correct at $16.42 per 1,000 pages, while Mistral OCR was best value at 258 of 264 for $2.02. On a faded receipt, GPT-6 Luna and GPT-6.1 Sol invented card digits, a server name and wifi passwords, errors that character error rates do not reveal. The post includes code to transcribe scans, turn reasoning off, cross-check two models from different labs and run Mistral OCR through the file-parser plugin.

Apple study evaluates LLM judges for personalized writing style assessment

Apple researchers tested how well LLM judges assess whether generated text matches a user's preferred writing style, generating text with four LLMs across three long-form writing tasks and comparing judge annotations against human pairwise labels. Judge quality correlated only weakly with general LLM ability on MMLU (Pearson's ρ ≤ 0.2) and varied by writing task and evaluation criteria, and paired comparisons proved more consistent than independent scoring. The strongest LLMs did not most reliably reproduce human evaluations, possibly due to exaggerated sensitivity to subtle details.

Ethan Mollick tests Claude Fable 5.2 across coding, business and research, calling it almost AGI

2 on tasks spanning hardware hacking, business planning, humanities course building, academic research and a 450-world 3D portal project, and judged it almost AGI under his definition of matching most human experts at most computer-based intellectual tasks.

What Frontier Labs Buy: A Data Deep Dive with Handshake AI's Garrett Lord and Team

Handshake's AI training business went from launch to $1 billion in gross annualized revenue in roughly a year, and this Latent Space deep dive with Garrett Lord and his team walks through the reinforcement learning environments frontier labs buy. In a finance demo, an agent working a simulated bank's email and deal files carries a goodwill figure into Python and subtracts it, a mistake that shows where expert judgment still beats the model. The team also presents VIALS, its benchmark for interpreting scientific images, and traces an agent's tool calls through safety simulations including CAREBench.

Google DeepMind and Artificial Analysis launch MKA benchmark for agentic knowledge work on complex visual files

Google DeepMind partnered with Artificial Analysis to launch the Multimodal Knowledge Agent (MKA) benchmark, which tasks agents with reviewing dense visual project files such as architectural drawings, engineering schematics and clinical scans to produce slide decks, spreadsheets, PDFs, Word documents and marked-up images.

Review: Gemini 4 Carbon's coding outpaces its judgment

Every's hands-on review finds Gemini 4 Carbon, released today by Google DeepMind, is a much stronger coder than Gemini 4 Argon and can build better 3D objects and work for hours with little supervision. Testers disagree on how much supervision it needs, and the review says Anthropic's Fable 5.2, released a day earlier, has raised expectations that a flagship agent also pick the right work, respect limits and know when it is finished. Google gave Every early access and had no input on the review.

Best AI Icon Generators in 2026: 7 Tools Compared

Recraft is named the best AI icon generator in 2026 after a seven-tool comparison that scored a five-icon consistent-style test and native SVG export. Recraft held stroke weight and corner radius across glyphs and exports editable SVG, while Canva passed only partially and Adobe Firefly and Midjourney failed both checks with raster output.

Best AI Meme Generators in 2026: 7 Tools Compared

Seven AI meme generators — Imgflip, Canva, Supermeme.ai, Recraft, Kapwing, ChatGPT and Magic Hour — were tested on free tier, watermark, templates, caption rendering, export sizes and API using the same three meme briefs. The comparison finds no universal winner: Imgflip wins for classic templates, Supermeme.ai for idea-to-caption, Recraft and ChatGPT for original images, Kapwing for video and GIF, and Canva when the meme must fit a layout.

Best AI Sticker Generators in 2026: 7 Tools Compared

Recraft is the best AI sticker generator for die-cut outlines, transparent PNGs, and SVGs from one canvas, while Canva is better for sticker sheets and print orders in the same editor. Seven tools were scored on November 9, 2026 against five print criteria, and only Recraft, Kittl, and Canva Print offer a real path to cut lines or print sheets. Firefly, Ideogram, Leonardo AI, and ChatGPT win narrower jobs such as photo-to-sticker conversion or chat packs.