HOPE: Higher-Order Pruning of Experts in Mixture-of-Experts Language Models

Researchers introduce HOPE, a second-order expert pruning method for Mixture-of-Experts language models that accounts for pairwise interactions between co-activated experts instead of scoring each expert independently. At 50% pruning, HOPE achieves the best average rank of any method, with gains of up to +6.1% on agentic coding tasks over the next-best pruning method. The work targets the memory bottleneck in models such as Qwen3.5-122B, where expert MLPs take up 95% of a ~250 GB footprint.