Apple-affiliated researchers distill Pythia-1B into Attention-free Mamba at perplexity 14.01

Apple-affiliated researchers published a cross-architecture distillation recipe that converts a pretrained Transformer into a Mamba-style state-space model with no Attention blocks. The authors are Abhinav Moudgil, Ningyuan Huang, Eeshan Gunesh Dhekane, Pau Rodríguez Lopez, Luca Zappella, and Federico Danieli. Moudgil and Huang did the work while at Apple, with affiliations also listed at Concordia and MILA, and at Johns Hopkins and the Flatiron Institute. Stage one distills the Transformer into linearized Attention via an adaptation of the kernel trick. Stage two maps that intermediate model into an adapted Mamba, using a principled initialization the authors say prior naive distillation lacked.

The reported language-modeling result is on Pythia-1B. The Transformer teacher scores perplexity 13.86. The distilled Mamba scores 14.01, a gap of 0.15. Ablations ran at 1B scale on 10 billion tokens. The authors varied the sequence mixer, scaled model size and distillation-token budget, and tested how tokens split across the two stages. Most downstream-task performance carries over. Gaps remain on tasks that depend more on in-context recall, where the model must retrieve a specific earlier item rather than continue probable text. The research-page summary does not list per-task scores.

A Transformer keeps a key-value cache that grows with context. Mamba keeps a fixed recurrent state. The paper cites lower memory use and higher generation throughput versus Attention. On-device inference is limited by memory bandwidth and energy per token. A 0.15 perplexity gap on Pythia-1B is small beside removing Attention, but average loss hides the failure the authors flag: in-context recall still lags. Prior cross-architecture distillation kept hybrid Attention-SSM blocks because a naive transfer lost the teacher. This recipe treats a principled initialization and a linearized-Attention bridge as the substitute for those blocks.

This is a research page, not a shipping-model note. No device, OS, or silicon identifier is attached to the distilled checkpoint. The useful next measurement is whether the recall gap closes as model size and distillation tokens move past the 1B and 10B-token ablations, and whether the token split between stages stays sensitive at those budgets. If a pure Mamba student stays within a few tenths of perplexity of an existing Transformer teacher, labs can convert checkpoints they already trained instead of pretraining SSMs from scratch. If recall does not close, hybrids remain the practical stack for context-heavy assistants.