Apple Researchers Derive Theoretical Limits of KL-Regularized LM Alignment
Start here
Researchers at Apple Machine Learning Research have derived the information-theoretic limits of KL-regularized language model alignment, giving a closed-form expression for the maximum expected reward gain achievable under a fixed KL budget. They report that this limit is governed by a Jeffreys divergence term rather than the √KL scaling used in prior analyses. The work also characterizes reward hacking: the gap between ideal and proxy alignment grows with reward error and with shrinking KL penalties, and the authors prove that reward ensembling mitigates it. On safety and summarization tasks, they find best-of-N closely approaches the theoretical limit, while PPO and GRPO remain substantially suboptimal.
Generated from reports · updated 13 h ago
Report timeline
Follow the reports to see the event from different sides.
- Apple Machine Learning ResearchTheoretical Limits of Language Model Alignment
Researchers from Apple Machine Learning Research derive the information-theoretic limits of KL-regularized language model alignment, giving a closed-form expression for maximum expected reward gain under a fixed KL budget that is governed by a Jeffreys divergence term rather than the √KL used in prior analyses. They show the gap between ideal and proxy alignment (reward hacking) grows with reward error and shrinking KL penalties, and prove reward ensembling mitigates it. On safety and summarization tasks, best-of-N closely approaches the theoretical limit while PPO and GRPO remain substantially suboptimal.
Heat for this event
Not enough continuous observations yet to draw a trend.