vLLM adds opt-in FlyDSL MonoKernel decode path for AMD GPUs
Start here
vLLM has merged an opt-in "mono decode" path, enabled with the --monokernel flag, that uses FlyDSL MonoKernels to fuse multiple dependent operator stages into a single persistent GPU kernel launch on AMD Instinct GPUs. The kernels reuse thread blocks across stages, prefetch weights and pass data through tagged mailboxes. Across the merged PRs for DeepSeek-V4.1-Flash, DeepSeek-V4-Pro, MiniMax-M3, Kimi-K3 and Muse Spark, vLLM reports 1.20x to 2.26x gains in time per output token at concurrency 1, the setting that matters for interactive agents; the gains shrink as concurrency rises, and each step is width-limited. Mono decode is opt-in: users add --monokernel to a supported recipe, and if a deployment is unsupported vLLM refuses to start and lists every reason. Ports for GLM-5.x, Qwen3.8 and MiniMax-M3.1 are still in review. Without scaled FP4/FP8 MFMA, the kernels keep their own copy of the weights, which for DeepSeek-V4.1-Flash at TP4 means 170 GiB per rank instead of 99 GiB. vLLM reports accuracy with mono on and off for each configuration.
Generated from reports · updated 29 min ago
Report timeline
Follow the reports to see the event from different sides.
- vLLM BlogvLLM uses FlyDSL MonoKernels to cut decode latency for interactive agents on AMD Instinct GPUs
vLLM has merged an opt-in mono decode path, enabled with --monokernel, that uses FlyDSL MonoKernels to combine multiple dependent operator stages into a single persistent GPU kernel launch on AMD Instinct GPUs. The approach reuses thread blocks across stages, prefetches weights and hands off data through tagged mailboxes, improving concurrency-1 interactivity or time per output token by 1.20x to 2.26x across merged PRs for DeepSeek-V4.1-Flash, DeepSeek-V4-Pro, MiniMax-M3, Kimi-K3 and Muse Spark. Gains shrink at higher concurrency, steps are width-limited, and accuracy is reported with mono on and off for each configuration.
Heat for this event
Not enough continuous observations yet to draw a trend.