vLLM Blog· Felix Li, Lingpeng Jin, Hattie Wu, Carlus Huang, Fangzhou Ai, Zaidy Ali, Amanzhol Salykov, Hyukjoon Lee, Ziming Huang, Jun Chen, Yifeng Huang, Bin Ding, Xiaobing Zhang, Yi Gan, Vinayak Gokhale, Hongxia Yang, Doug Lehr, Richard Li, Andy Luo, Peng Sun, and Emad Barsoum·· 15 h agoAI score58
vLLM uses FlyDSL MonoKernels to cut decode latency for interactive agents on AMD Instinct GPUs
Keeping Agents Fast: High-Interactivity Decode with FlyDSL MonoKernels in vLLM
AI summary
vLLM has merged an opt-in mono decode path, enabled with --monokernel, that uses FlyDSL MonoKernels to combine multiple dependent operator stages into a single persistent GPU kernel launch on AMD Instinct GPUs. The approach reuses thread blocks across stages, prefetches weights and hands off data through tagged mailboxes, improving concurrency-1 interactivity or time per output token by 1.20x to 2.26x across merged PRs for DeepSeek-V4.1-Flash, DeepSeek-V4-Pro, MiniMax-M3, Kimi-K3 and Muse Spark. Gains shrink at higher concurrency, steps are width-limited, and accuracy is reported with mono on and off for each configuration.
Source:vLLM Blog · vllm.ai