Rapid LLM development is outrunning the research meant to explain it, study finds

| Source: THE DECODER

Tags: Stanford, James Zou, Owen Queen, sycophancy, LLM-as-a-judge, lost in the middle

Stanford researchers retested 50 widely cited LLM findings on 20 current models and found 44% have expired and 34% weakened, with an average lifespan of 15.7 months for dated claims.

Details

Owen Queen and James Zou checked 50 claims about large language models against 20 models released between July 2024 and September 2026 by eight developers. Only 16% of findings still hold and 6% got stronger. Among 22 findings with an estimated expiration date, average lifespan was 15.7 months; 16% expired after one year and 39% after two. Claims about what models cannot do expire fastest. Five of nine reasoning findings are expired, including letter counting in strawberry, lost-in-the-middle context and caving to user pushback. Findings on sycophancy, LLM-as-evaluator bias and political lean proved more durable. For practitioners, older capability limitations should be re-verified on current models before they drive design decisions.