The Symmetry That Breaks Neural Network Averaging

September 9, 2026 | Source: Towards Data Science

Tags: weight averaging, model merging, permutation symmetry, federated learning, neural networks, LLM engineering

Permutation symmetry explains why averaging the weights of two identically-trained neural networks almost always produces a worse model: each network assigns the same functions to different neurons, so naive weight averaging mixes incompatible representations — a root cause of failures in model merging and federated learning.

Details

When two neural networks are trained on the same data with the same architecture but different random seeds, both converge to similar loss values — but not to the same parameter configuration. The reason is permutation symmetry: the order of hidden neurons is arbitrary, and swapping entire columns of weights produces a functionally equivalent network. This means the two networks may compute identically while having their internal representations arranged in completely different orders. Naive weight averaging treats these differently-ordered parameters as if they were in correspondence. The result is roughly equivalent to averaging the revenue column in one spreadsheet with the headcount column in another — the numbers combine without meaning. The averaged network typically performs substantially worse than either parent. This structural property explains two important practical phenomena in 2026 LLM engineering. First, model merging (combining fine-tuned variants of the same base model) requires alignment steps like SLERP, Git Re-Basin, or permutation-matching algorithms before averaging can work. Second, federated averaging across clients trained on heterogeneous data faces the same misalignment problem by default. The article is an explainer rather than a new research result, but it builds from solid mathematical foundations and is well-suited for practitioners who want to understand why their model soup experiments fail.