Skip to content
Hot eventDeveloping

vLLM Team Publishes Guide to Disaggregated Serving

1 reports1 reporting sourcesUpdated 16 h ago

Start here

AI digest

The vLLM team published a practical guide to disaggregated serving in vLLM v0.31.0 and later, splitting prefill, decode and CPU-side rendering into separate services. In a test on two NVIDIA L40S GPUs with Qwen2.5-7B-Instruct and ~8k-token prompts, collocated p99 inter-token latency rose from 23 ms to 169 ms at 0.4 req/s, while the prefill/decode setup stayed between 25 and 52 ms. The guide warns that slow KV-cache transfer affects every first token: on the test machine, which lacked peer-to-peer copies, moving each 8k prompt's ~470 MB cache took about 1.3 s. The guide covers when each split is worth using and how to run it; the proxy in tests/ is described as suitable for development but not production.

Generated from reports · updated 16 h ago

Report timeline

Follow the reports to see the event from different sides.

November 19
  1. Hacker News · AI
    vLLM guide explains disaggregated serving for prefill, decode and CPU rendering

    The vLLM team published a practical guide to disaggregated serving in vLLM v0.31.0 and later, splitting prefill, decode and CPU-side rendering into separate services. On two NVIDIA L40S GPUs with Qwen2.5-7B-Instruct and ~8k-token prompts, collocated p99 inter-token latency jumped from 23 ms to 169 ms at 0.4 req/s while the prefill/decode setup stayed between 25 and 52 ms. The guide warns that slow KV-cache transfer lands on every first token, and on the test box without peer-to-peer copies each 8k prompt's ~470 MB cache took about 1.3 s to move.

Heat for this event

Current, comparable range
6
Peak, comparable range
10November 19 12:00
Change in 24 hours
–

The trend compares the same participants over hours observed continuously, so its range can be narrower than the current heat count. Hover or tap the chart for hourly heat; the arrow keys move through it.