Apple Machine Learning Research·· 3 d agoAI score38
Apple Study: Gating Mechanism Explains Why SSMs Memorize Instead of Learning In-Context
On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models
AI summary
An Apple and Harvard study shows that the gating mechanism in State Space Models (SSMs) causes them to first learn an in-weights "memorization" solution, delaying or even preventing convergence to correct in-context learning. This occurs even when the architecture and memory capacity impose no fundamental limits. Gating, however, often improves generalization to long sequence lengths.
Source:Apple Machine Learning Research · machinelearning.apple.com