Attention to Mamba: A Recipe for Cross-Architecture Distillation

Apple researchers propose a two-stage recipe for distilling a Transformer into a pure Mamba architecture without any Attention blocks. The method first distills a pretrained Transformer into a linearized Attention model via an adapted kernel trick, then distills that into an adapted Mamba model. The resulting distilled Mamba retains most of Pythia-1B's performance, reaching 14.01 perplexity versus the teacher's 13.86, with some gaps on tasks relying on in-context recall.