vLLM Team Publishes Guide to Disaggregated Serving
Start here
The vLLM team published a practical guide to disaggregated serving in vLLM v0.31.0 and later, splitting prefill, decode and CPU-side rendering into separate services. In a test on two NVIDIA L40S GPUs with Qwen2.5-7B-Instruct and ~8k-token prompts, collocated p99 inter-token latency rose from 23 ms to 169 ms at 0.4 req/s, while the prefill/decode setup stayed between 25 and 52 ms. The guide warns that slow KV-cache transfer affects every first token: on the test machine, which lacked peer-to-peer copies, moving each 8k prompt's ~470 MB cache took about 1.3 s. The guide covers when each split is worth using and how to run it; the proxy in tests/ is described as suitable for development but not production.
Generated from reports · updated 16 h ago
Report timeline
Follow the reports to see the event from different sides.
- Hacker News · AIvLLM guide explains disaggregated serving for prefill, decode and CPU rendering
The vLLM team published a practical guide to disaggregated serving in vLLM v0.31.0 and later, splitting prefill, decode and CPU-side rendering into separate services. On two NVIDIA L40S GPUs with Qwen2.5-7B-Instruct and ~8k-token prompts, collocated p99 inter-token latency jumped from 23 ms to 169 ms at 0.4 req/s while the prefill/decode setup stayed between 25 and 52 ms. The guide warns that slow KV-cache transfer lands on every first token, and on the test box without peer-to-peer copies each 8k prompt's ~470 MB cache took about 1.3 s to move.
Heat for this event
- Current, comparable range
- 6
- Peak, comparable range
- 10November 19 12:00
- Change in 24 hours
- –
The trend compares the same participants over hours observed continuously, so its range can be narrower than the current heat count. Hover or tap the chart for hourly heat; the arrow keys move through it.