vLLM runs DeepSeek-V4.1-Flash on Vera Rubin NVL72 at 58x GB300 throughput
Start here
vLLM reports it served DeepSeek-V4.1-Flash on a single NVIDIA Vera Rubin NVL72 rack at 1.7M tokens/s per GPU and 387 tokens/s per user, which it says is 58x the throughput of GB300 NVL72. Peak figures were 2.52M tokens/s rack-wide and 1,810 tokens/s for a single user. According to vLLM, the gains come from kernels rebuilt for Rubin's split memory domains and FP4 tensor cores, a specialized TP8 decode path, and rack-wide KV and Engram pools shared over NVLink. With prefixes pooled across the rack and Dynamo workload-aware routing, vLLM says the prefix hit rate holds at 96.5–97% even at high concurrency.
Generated from reports · updated 12 h ago
Report timeline
Follow the reports to see the event from different sides.
- vLLM BlogPicksvLLM serves DeepSeek-V4.1-Flash on Vera Rubin NVL72 at 58x GB300 NVL72 throughput
vLLM reports serving DeepSeek-V4.1-Flash on a single NVIDIA Vera Rubin NVL72 rack at 1.7M tokens/s per GPU with 387 tokens/s per user, 58x the GB300 NVL72, peaking at 2.52M tokens/s and 1,810 tokens/s for a single user. The gains come from kernels rebuilt for Rubin's split memory domains and FP4 tensor cores, a specialized TP8 decode path, and rack-wide KV and Engram pools shared over NVLink. With prefixes pooled across the rack and Dynamo workload-aware routing, the prefix hit rate holds at 96.5-97% even at high concurrency.
Heat for this event
Not enough continuous observations yet to draw a trend.