Skip to content
Hacker News · AI·· 16 h agoAI score60

vLLM guide explains disaggregated serving for prefill, decode and CPU rendering

Taking vLLM Apart: A Practical Guide to Disaggregated Serving

AI summary

The vLLM team published a practical guide to disaggregated serving in vLLM v0.31.0 and later, splitting prefill, decode and CPU-side rendering into separate services. On two NVIDIA L40S GPUs with Qwen2.5-7B-Instruct and ~8k-token prompts, collocated p99 inter-token latency jumped from 23 ms to 169 ms at 0.4 req/s while the prefill/decode setup stayed between 25 and 52 ms. The guide warns that slow KV-cache transfer lands on every first token, and on the test box without peer-to-peer copies each 8k prompt's ~470 MB cache took about 1.3 s to move.

Source:Hacker News · AI · vllm.ai