The Symmetry That Breaks Neural Network Averaging

Naive weight averaging of neural networks often fails because of a structural property called permutation symmetry. This symmetry arises because the order of neurons in a hidden layer is arbitrary, creating many functionally identical but parametrically different solutions in the loss landscape. Averaging the weights of two models that have converged to different permutations is like averaging misaligned spreadsheet columns, resulting in a high-loss, suboptimal model. This problem can be overcome by first algorithmically aligning the neurons between models before averaging, a process that enables successful model merging. Beyond permutation, other symmetries like positive-rescaling also contribute to the complexity of the neural network loss landscape and must be considered.