The Channel-Wise Attention | Squeeze and Excitation

Squeeze and Excitation Networks (SENet) introduce a channel-wise attention mechanism for computer vision, unlike the spatial attention found in Vision Transformers. The core module operates in two steps: a "squeeze" phase that uses global average pooling to capture channel-wise statistics, and an "excitation" phase that uses small neural networks to learn the importance of each channel. These learned importance weights are then used to rescale the feature maps, amplifying informative channels and suppressing less useful ones. SENets are designed as lightweight building blocks that can be integrated into existing CNN architectures like ResNet to improve performance. Adding these modules has been shown to significantly boost model accuracy with only a marginal increase in computational cost.