Skip to content
Hot eventDeveloping

vLLM adds opt-in FlyDSL MonoKernel decode path for AMD GPUs

1 reports1 reporting sourcesUpdated 15 h ago

Start here

AI digest

vLLM has merged an opt-in "mono decode" path, enabled with the --monokernel flag, that uses FlyDSL MonoKernels to fuse multiple dependent operator stages into a single persistent GPU kernel launch on AMD Instinct GPUs. The kernels reuse thread blocks across stages, prefetch weights and pass data through tagged mailboxes. Across the merged PRs for DeepSeek-V4.1-Flash, DeepSeek-V4-Pro, MiniMax-M3, Kimi-K3 and Muse Spark, vLLM reports 1.20x to 2.26x gains in time per output token at concurrency 1, the setting that matters for interactive agents; the gains shrink as concurrency rises, and each step is width-limited. Mono decode is opt-in: users add --monokernel to a supported recipe, and if a deployment is unsupported vLLM refuses to start and lists every reason. Ports for GLM-5.x, Qwen3.8 and MiniMax-M3.1 are still in review. Without scaled FP4/FP8 MFMA, the kernels keep their own copy of the weights, which for DeepSeek-V4.1-Flash at TP4 means 170 GiB per rank instead of 99 GiB. vLLM reports accuracy with mono on and off for each configuration.

Generated from reports · updated 29 min ago

Report timeline

Follow the reports to see the event from different sides.

November 18
  1. vLLM Blog
    vLLM uses FlyDSL MonoKernels to cut decode latency for interactive agents on AMD Instinct GPUs

    vLLM has merged an opt-in mono decode path, enabled with --monokernel, that uses FlyDSL MonoKernels to combine multiple dependent operator stages into a single persistent GPU kernel launch on AMD Instinct GPUs. The approach reuses thread blocks across stages, prefetches weights and hands off data through tagged mailboxes, improving concurrency-1 interactivity or time per output token by 1.20x to 2.26x across merged PRs for DeepSeek-V4.1-Flash, DeepSeek-V4-Pro, MiniMax-M3, Kimi-K3 and Muse Spark. Gains shrink at higher concurrency, steps are width-limited, and accuracy is reported with mono on and off for each configuration.

Heat for this event

Not enough continuous observations yet to draw a trend.