REINFORCE Zero: Kernel Alignment Enables Stable Async RL at Policy Lag 64, Extends to DeepSeek-V4.1 and Kimi K3.1
Zero Mismatch, Simple Async RL: REINFORCE Zero, DeepSeek-V4.1 and Kimi K3.1
NVIDIA Research and Inferact's REINFORCE Zero paper shows that aligning rollout and trainer kernels lets unclipped token-level importance sampling correct the remaining weight lag, training stably for 20,000 updates at policy lag 64 and reaching 2.40x the training-step throughput of synchronous training with aligned kernels. With standard kernels, the median rollout-to-learner KL rises 83.33x across the run and online AIME24 pass@1 falls from 34.27% to 13.96%, while a sequence gate stabilizes training but rejects 24.48% of sequences by the last 1,000 updates. The zero-mismatch setup now covers DeepSeek-V4.1-Flash, Kimi K3.1 on release day, and DeepSeek-V4-Flash and Kimi K3.1 on AMD MI355X, with code in Inferact/zero-mismatch-rl and NVIDIA-NeMo/labs-molt.
NVIDIA Research and Inferact split the async RL rollout-learner gap into weight lag and kernel mismatch, showing that aligning kernels lets unclipped importance sampling hold stable training at policy lag 64.
Source:vLLM Blog · vllm.ai