Explicit State Elicitation Is Not Enough: A Controlled Audit of Memory-Policy Classification

Details

Personalized agents need to decide whether retrieved user memory should be used, ignored, updated, or queried — a routing problem where structured intermediate outputs seemed promising. This paper runs a careful empirical audit to test whether explicit state-output fields actually improve routing decisions. The authors first caught a dataset shortcut: a 480-example synthetic development set initially showed large gains from state-structured prompts, but TF-IDF diagnostics revealed the improvement came from lexical separability (no positive standalone Ignore cases), not genuine reasoning. They then constructed a frozen 160-example controlled counterfactual set with 40 matched four-way families. On this cleaner set, exposing the four state definitions improves accuracy, but an isolated explicit state-output field does not significantly improve policy accuracy for Llama-3.3-70B and gives only a marginal, non-significant gain for GPT-OSS-120B. Supplying benchmark-associated state labels shifts predictions — but because those labels deterministically map to policies, this is a label-conditioning diagnostic, not evidence of faithful reasoning. Family-level analysis shows that complete four-way success is rare, meaning single-example accuracy overstates counterfactual consistency. For agents relying on memory-policy classification, this is a useful negative result: simple structured prompting is not enough.