Sakana AI Releases JFinBench Japanese Financial Benchmark
Start here
Sakana AI released JFinBench, a Japanese financial benchmark built from six 2026 financial certification exams totaling 321 questions, including written and image-based items, and published the dataset, code and leaderboard. Across 12 LLMs, Claude Fable 5 scored 95.8 and GPT-6 Astra 93.6; the best open-weight model, Kimi K3.1 Pro, reached 83.4, about 10 points behind GPT-6 Astra despite reportedly matching it on coding benchmarks. Sakana AI says most failures came from missing knowledge of Japanese regulations rather than calculation errors. The benchmark was built with a human-in-the-loop agentic pipeline that took about 12 hours of human verification.
Generated from reports · updated 12 min ago
Report timeline
Follow the reports to see the event from different sides.
- Sakana AI BlogSakana AI releases JFinBench, a Japanese financial benchmark built from 2026 certification exams
Sakana AI released JFinBench, a Japanese financial benchmark built from six 2026 financial certification exams totaling 321 questions, including written and image-based items. Across 12 LLMs, Claude Fable 5 scored 95.8 and GPT-6 Astra 93.6, while the best open-weight model, Kimi K3.1 Pro, reached 83.4, about 10 points behind GPT-6 Astra despite reportedly matching it on coding benchmarks. Most failures came from missing knowledge of Japanese regulations rather than calculation errors, and the benchmark was built with a human-in-the-loop agentic pipeline that took about 12 hours of human verification.
Heat for this event
Not enough continuous observations yet to draw a trend.