Amazon Web Services researchers propose priming to convert pretrained Transformers into hybrid state space models without full retraining
Amazon Web Services researchers introduce priming, a method that reuses pretrained Transformer weights to cut the pretraining cost of hybrid state space architectures by more than 99%, using under 0.5% of the source model's pretraining token budget. In controlled comparisons, Gated KalmaNet outperformed Gated DeltaNet and Mamba-2, and a 32B GKA-based hybrid matched its source Transformer on reasoning benchmarks while decoding up to twice as quickly at 128,000 tokens of context. The team released 8B and 32B primed hybrid models, training code, Triton kernels, and a vLLM serving plugin under the Apache 2.0 License.
Why it matters: Amazon Web Services researchers show that reusing pretrained Transformer weights can cut hybrid architecture pretraining cost by over 99%, making controlled comparisons of state space layers practical.