Mixture of Experts (MoEs) in Transformers

Mixture of Experts (MoEs) provide an alternative to scaling dense language models by replacing certain feed-forward layers in a Transformer with multiple "expert" sub-networks. A router mechanism dynamically selects a small subset of these experts to process each token, enabling sparse activation of the model's parameters. This architecture allows models to have a very large number of total parameters for high capacity while only activating a fraction for inference, leading to much faster performance and lower computational costs. This approach provides better compute efficiency, a natural axis for parallelization, and has been widely adopted in recent models to overcome the practical limits of dense scaling.