Google Research's sleep framework lets LLMs consolidate new knowledge without catastrophic forgetting

Google Research introduced a sleep framework for continual learning that alternates active interaction with offline memory consolidation, letting LLMs absorb new facts into expanded Mixture-of-Experts parameters instead of losing prior knowledge. The approach combines parameter expansion, on-policy distillation with reinforcement learning-based imitation, and a dreaming stage that generates and filters synthetic training samples. The authors report near-zero performance drop in continual translation for unseen languages such as Manchu and Kalamang, near-perfect scaling to 10M tokens on BABILong, 48.9% single-task and 46.2% CPT results on SQuAD-based factual incorporation, and 79.2%, 69.0% and 46.1% on AIME 24/25 and HMMT 25.