Ai2 study tests whether LLMs can reliably judge research idea novelty

Ai2 released Old Ideas, Novel Problems, a study and benchmark testing how reliably LLMs evaluate the novelty of research ideas, along with data, code and a pipeline for building new evaluation sets. Across six judges including GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6, small prompt changes shifted accuracy by more than 50 percentage points on AI-generated ideas, and a simple prompted Claude Opus 4.6 baseline scored 0.854 macro-F1 at $2.59 versus 0.642 at $30.66 for Idea Novelty Checker and 0.772 at $82.05 for Sakana AI's AI Scientist.