MoNe: Modular Neural Memory for Efficient Long Context Inference

August 19, 2026 | Source: arXiv AI

Tags: long-context inference, Transformer, KV cache, memory efficiency, MoNe, GPU optimization

MoNe attaches to any frozen pretrained Transformer as a lightweight plug-in, enabling 128K-token inference with ~80% reduction in both compute and peak GPU memory compared to in-context learning, while achieving O(1) query cost regardless of context length — no retraining required.

Details

Long-context inference with standard Transformers scales quadratically in compute and linearly in peak GPU memory — making 128K+ token contexts expensive or impractical. MoNe (Modular Neural Memory) addresses this without touching the underlying model: it reads context in fixed-size segments via test-time learning, generating keys and values from query tokens alone at inference without re-reading context tokens. The design has two phases: preprocessing reads the full context once in segments to build fast-weight memory networks; inference queries these memories in O(1) time per token, with peak GPU memory that does not grow with context length N. This decouples inference cost from context length: O(N) preprocessing and O(1) query cost. At 128K tokens, MoNe reduces both compute and peak GPU memory by approximately 80% compared to standard in-context learning (ICL), with only 6.4% parameter overhead. On RULER benchmarks (needle-in-a-haystack, word extraction), MoNe achieves strong performance even at lengths far beyond the backbone's native context window, while ICL degrades sharply. For practitioners running long-context inference on GPU-constrained infrastructure, MoNe is directly applicable — it attaches to frozen models without retraining, making it immediately usable with existing checkpoints. The 80% memory reduction at 128K tokens translates to roughly 4-5× more concurrent sessions per GPU or the ability to run models previously requiring H100s on smaller hardware.