Dead Weights, Live Signals: Feedforward Graphs of Frozen Language Models

September 14, 2026 | Source: arXiv AI

Tags: frozen LLMs, model composition, Llama, Qwen, Gemma, Mistral, LLM architecture, multi-model

Researchers achieve 87.3% on ARC-Challenge using a graph of five frozen LLMs connected by 17.6M trainable linear projections—outperforming the best single constituent model by 11.4 points—demonstrating that backpropagation through frozen model boundaries is tractable and that emergent routing behavior appears without explicit supervision.

Details

Building on earlier findings about geometric compatibility between independently trained LLM latent spaces, this paper extends the concept from two-model static steering to end-to-end trainable multi-node graphs of frozen models. Three small frozen models (Llama-3.2-1B, Qwen2.5-1.5B, Gemma-2-2B) encode input into a shared latent space via learned linear projections, whose aggregate signal is injected into two larger frozen models (Phi-3-mini, Mistral-7B) through residual stream hooks; a lightweight cross-attention output node produces the final result.\n\nWith only 17.6M trainable parameters against approximately 12B frozen, the architecture achieves 87.3% on ARC-Challenge, 82.8% on OpenBookQA, and 67.2% on MMLU—outperforming the best single constituent model by 11.4, 6.2, and 1.2 percentage points respectively. Gradient flow through multiple frozen model boundaries is empirically verified to be tractable.\n\nPerhaps the most interesting finding is emergent behavior: the output node develops selective routing across the two larger frozen models without explicit supervision—the system learns which 'large' model to rely on for which inputs. This suggests frozen model graphs can develop specialization organically.\n\nFor practitioners, this opens the possibility of assembling capable inference-time systems from existing frozen checkpoints without expensive fine-tuning, though scaling and memory costs need further investigation.