ข้ามไปที่เนื้อหา

ประเภทเนื้อหา

เบนช์มาร์ก ล่าสุด

โมเดลไหนแข็งกว่ากันจริง: คะแนนเบนช์มาร์ก ข้อถกเถียงเรื่องวิธีประเมิน และการขยับของอันดับ

8 รายการคัดสรร6 รายการใน 30 วันที่ผ่านมาเก็บไว้ 86 รายการ

อัปเดต 13 พฤศจิกายน 00:51

เบนช์มาร์ก

วันนี้13 พฤศจิกายนวันศุกร์รายการ 1–8 จากทั้งหมด 8
  1. AWS Machine Learning Blog60

    AWS open-sources UAEF, a framework-agnostic evaluation system for AI agents

    AWS has open-sourced UAEF (Universal Agent Evaluation Framework), a framework-agnostic system that uses adapters to normalize agent traces from LangGraph, LangChain, Strands Agents, Amazon Bedrock AgentCore and Langfuse into one canonical AgentTrace, then scores them with 36 built-in metrics across 7 dimensions. Results are stored per experiment in Amazon DynamoDB and Amazon S3, so teams can compare agent versions against a baseline, flag regressions with a default 5 percent threshold, and generate statistical reports plus LLM-written insights. The post says customers across 12 organizations in telecom, healthcare and financial services used UAEF before release, and the Apache 2.0 code, Python library, CDK deployment and per-framework notebooks are in the awslabs/uaef GitHub repository.

    ทำไมเรื่องนี้สำคัญ: The AWS team has open-sourced UAEF, which uses adapters to unify agent trajectories from different frameworks into a single format, then scores them across 7 dimensions using 36 metrics, providing a reusable evaluation layer for comparing agent versions across frameworks.

11 พฤศจิกายนวันพุธ
  1. OpenRouter Announcements60

    OpenRouter tests what fp8 and fp4 quantization do to LLM quality and price

    OpenRouter ran 280 exact-answer tasks against six endpoints of Qwen3.8 27B on October 20, 2026 and did not detect an accuracy difference between fp16, fp8 and fp4, with all six scoring between 98.3% and 99.2%. Across 1,455 endpoints, 59.5% report no precision at all and fp8 is the most common labeled format at 23.6%. Lower precision was cheaper for most models but not all, and provider choice moved price more than precision did, with the priciest same-precision endpoint a median 1.72 times the cheapest.

    ทำไมเรื่องนี้สำคัญ: OpenRouter measured fp8, fp4 and fp16 endpoints of one model across 240 exact-answer tasks and found no detectable accuracy gap, while price differences tracked the provider more than the precision.

10 พฤศจิกายนวันอังคาร
  1. Latent Space85

    Anthropic releases Claude Fable 5.2 and Mythos 5.2, retaking the top LLM spot as its IPO roadshow begins

    Anthropic released Claude Fable 5.2 and Claude Mythos 5.2, which top most independent leaderboards, including Artificial Analysis Intelligence Index v5 at 65 and SemiAnalysis GDPx at 16.4%. Pricing drops 20% to $8/$40 per 1M input/output tokens, but SemiAnalysis reports cost per GDPx task rose from $67.16 to $88.15 because the model writes hundreds to thousands of test cases and spawns subagents to check its work. Multiple evals show the max effort level underperforming high, and Fable is capped at 50% of a weekly subscription limit.

    ทำไมเรื่องนี้สำคัญ: Anthropic releases Fable 5.2 and Mythos 5.2 with a 20% per-token price cut, yet independent evals show higher per-task cost and a max-effort setting that underperforms high.

  2. OpenRouter Announcements72

    OpenRouter benchmarks 15 vision models and Mistral OCR on scanned documents

    OpenRouter tested 15 vision models, Mistral OCR and Tesseract on eight self-made degraded scans, and Claude Opus 5.5 was most accurate with 262 of 264 key values correct at $16.42 per 1,000 pages, while Mistral OCR was best value at 258 of 264 for $2.02. On a faded receipt, GPT-6 Luna and GPT-6.1 Sol invented card digits, a server name and wifi passwords, errors that character error rates do not reveal. The post includes code to transcribe scans, turn reasoning off, cross-check two models from different labs and run Mistral OCR through the file-parser plugin.

    ทำไมเรื่องนี้สำคัญ: A hands-on benchmark of 15 vision models plus Mistral OCR and Tesseract on eight degraded scans, with runnable code for cross-checking two models to catch fabricated values.

  3. One Useful Thing78

    Ethan Mollick tests Claude Fable 5.2 across coding, business and research, calling it almost AGI

    2 on tasks spanning hardware hacking, business planning, humanities course building, academic research and a 450-world 3D portal project, and judged it almost AGI under his definition of matching most human experts at most computer-based intellectual tasks.

    ทำไมเรื่องนี้สำคัญ: Ethan Mollick's hands-on tests of Claude Fable 5.2 show strong general capability across coding, business and research, while exposing a persistent gap in self-judging when work is good enough.

28 ตุลาคมวันพุธ
  1. Artificial Analysis Articles82

    Gemini 4 Carbon tops Artificial Analysis Intelligence Index v5, ranks second on Coding Agent Index

    Google's Gemini 4 Carbon is the new No. 5 on the Artificial Analysis Coding Agent Index. 5 (max, 62), and it leads 4 of the 9 Index evaluations including HLE-Diamond and Terminal-Bench-Science. 1 rises from 4% to 11%; the model is being rolled out to selected users and is not yet publicly available.

    ทำไมเรื่องนี้สำคัญ: Gemini 4 Carbon tops Artificial Analysis Intelligence Index v5 at 59 and trails Claude Opus 5.5 by 3 points on the Coding Agent Index, with per-task cost and hallucination figures for comparison.

28 กันยายนวันจันทร์
  1. Anthropic · Claude.dev72

    Anthropic adds build-eval and hillclimb commands to the claude-api skill in Claude Code

    Anthropic added two commands to the claude-api skill inside Claude Code: /claude-api build-eval builds an evaluation set in your codebase, and /claude-api hillclimb iterates on an application against that evaluation with a held-out set to catch overfitting.

    ทำไมเรื่องนี้สำคัญ: Anthropic adds build-eval and hillclimb commands to its claude-api skill, showing how Claude Code can design evaluations and iterate against them while guarding against overfitting.

5 สิงหาคม 2024วันจันทร์
  1. Modal Blog62

    Modal replicates Large Language Monkeys: LLaMA 3.1 8B matches GPT-4o on HumanEval via search

    Modal's Charles Frye replicated the core claim of the Large Language Monkeys paper on a new dataset, showing that repeated sampling lifts LLaMA 3.1 8B from 66.4% pass@1 to 90.5% pass@100 and 95.1% pass@1000 on HumanEval, versus GPT-4o's reported 90.2% pass@1. The runs used vLLM on Modal serverless GPUs and Modal Sandboxes for parallel test execution, cost well under $50, and showed smooth, predictable scaling across three orders of magnitude.

    ทำไมเรื่องนี้สำคัญ: A hands-on replication shows how repeated sampling with a small open model can approach GPT-4o on HumanEval, offering a concrete, low-cost recipe for trading compute for accuracy.