Your Experts Have Partners. There's HOPE for Keeping Them Together

Mixture-of-Experts (MoE) models are incredibly large, and pruning—permanently deleting less useful "expert" subnetworks—is a common technique to make them more efficient. However, existing methods make a critical mistake by evaluating each expert in isolation, failing to account for the cooperative groups they form when processing information. A new second-order method called HOPE (Higher-Order Pruning of Experts) considers the crucial pairwise interactions between experts to make smarter pruning decisions. By preserving these cooperative partnerships, HOPE significantly outperforms other techniques with negligible extra computational cost, especially at aggressive pruning rates of 50%.