Theoretical Limits of Language Model Alignment

Researchers from Apple Machine Learning Research derive the information-theoretic limits of KL-regularized language model alignment, giving a closed-form expression for maximum expected reward gain under a fixed KL budget that is governed by a Jeffreys divergence term rather than the √KL used in prior analyses. They show the gap between ideal and proxy alignment (reward hacking) grows with reward error and shrinking KL penalties, and prove reward ensembling mitigates it. On safety and summarization tasks, best-of-N closely approaches the theoretical limit while PPO and GRPO remain substantially suboptimal.