Ai2 releases StateBench to test how language models track state changes across long documents

Ai2 introduced StateBench, a benchmark with four difficulty levels for long-context state tracking, from immutable assignment to self-referential and cross-referential updates. In tests, Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna handled the easier levels but dropped to 13% or lower on the longest general finite problems, while 275M-parameter Gated DeltaNet and hybrid models trained from scratch generalized almost perfectly on the first three levels and transformers never learned the hardest one.