Zyphra Research: Expert Coupling Cuts MoE Pretraining Communication

Zyphra Research shows MoE routing is highly structured, with co-selected experts correlated within a layer and a token's expert choices predicting later-layer routing. Exploiting this via correlated expert placement and token shuffling cuts all-to-all communication time up to 2.63× and improves end-to-end training step time up to 1.41× in Megatron-LM experiments on MI300X GPUs, without changing architecture or routing decisions.