AI systems quietly drop user instructions when they compress context

Details

When AI systems run out of context window space, they summarize past conversation history to free up room — a process called compaction. Penn State researchers systematically measured what gets lost, specifically tracking 'session constraints': user-imposed rules like 'never use my name' or 'confirm before taking any action.' Their findings are stark: on average, only 17% of user-injected session constraints survive compression. Most tested compactor systems actually perform worse than no compression at all. Even targeted prompts designed specifically to preserve user constraints keep retention below 40%. The security implications are direct. After compaction, an agent told 'don't send emails without my approval' may proceed freely — because that instruction was discarded. The researchers introduce COMPINT, a new evaluation suite, to measure this problem systematically. One finding stands out: GPT-5.4-mini beats baseline compactors in some scenarios, though details on why are limited. Their proposed fix is a small add-on LLM built on Qwen3.5-9B that plugs into the compression pipeline and preserves over 90% of session constraints without modifying the base model. One important caveat: even with the full uncompressed context, base model rule compliance sits at only 59–71%, meaning compression compounds an already imperfect foundation.