NeurIPS 2025 Best Paper Review: Qwen’s Systematic Exploration of Attention Gating

A NeurIPS 2025 best paper from the Qwen team systematically explores applying Gated Attention to Large Language Models. Gating is a signal modulation mechanism that uses a learned filter to selectively control the flow of information within a neural network, a concept historically used in LSTMs. The paper's primary contribution is identifying the optimal placement for a gate within the transformer's attention block. Placing a simple element-wise sigmoid gate immediately after the scaled dot-product attention operation yields the best results. This specific configuration leads to enhanced training stability, allows for the use of larger learning rates, and improves the model's overall scaling properties.