StateBench: Testing how language models track state changes over long documents

| Source: Hugging Face Blog

Tags: Ai2, StateBench, Gated DeltaNet, state tracking, long-context benchmarks, hybrid architectures

Ai2 released StateBench, a four-level benchmark showing recurrent and hybrid models generalize state tracking far better than transformers, while frontier models still fail on harder cases as inputs grow longer.

Details

Ai2 introduced StateBench, a benchmark for long-context state tracking: how well language models keep track of entities as they change across long documents, from reassigned variables in code to objects changing hands in a story. The benchmark has four difficulty levels, ranging from simply finding the latest update to following chains of dependent changes where one missed step breaks the answer. Ai2 reports that recurrent and hybrid architectures such as Gated DeltaNet generalize much farther than transformers when trained directly on these tasks. Frontier models including Gemini 3.8 Flash, Claude Opus 5.5 and GPT-6 Sol and Luna still break down on harder state tracking as inputs get longer. A tech report, dataset and code are released alongside the blog post.