Anchor: Augmented Cross-Layer Recurrent State for Long-Context Retrieval in Test-Time Training

| Source: arXiv AI

Tags: Test-Time Training, sliding-window attention, LoLCATs, Llama-3.1-8B, Mistral-7B, linear attention

Anchor places test-time-training memory only at a few anchor layers with a shared cross-layer state, giving 2.3-3.1x higher recall than LoLCATs on Llama-3.1-8B and Mistral-7B with a 10x smaller recurrent state.

Details

Linearizing a pretrained Transformer by mixing sliding-window attention with a constant-size recurrent memory makes inference cheaper, but long-context retrieval suffers. The authors studied an SWA plus Test-Time Training hybrid and found retrieval concentrates in a few anchor layers. Anchor puts TTT memory only at those layers and uses one cross-layer recurrent state written once and read by many layers, each through its own learned read map. On Llama-3.1-8B and Mistral-7B it reports 2.3-3.1x higher recall and 6.7-8.9 points higher MMLU than LoLCATs with a 10x smaller recurrent state, keeps 91-95% of the teacher's MMLU accuracy, and cuts end-to-end latency by up to 4.3x versus softmax attention.