vLLM adds day-0 support for Thinking Machines' Inkling 2 Small
Start here
vLLM added day-0 inference support for Thinking Machines' Inkling 2 Small, a multimodal text-and-vision mixture-of-experts model with 594B total and 17B active parameters and up to 1M tokens of context. vLLM says the model can be served on a single node. The integration adds kernel fusion with PDL, sequence parallelism for prefill that vLLM says cuts an 8K-token prefill step by 22% at TP4 and 26% at TP8, and a Gluon relative attention decode kernel that reduces a batch-1 decode step at 100K context from 15.1 ms to 4.8 ms. vLLM reports up to 630 TPS/user at TP4 on GB200 with acceptance length 3.5, and says LoRA and reinforcement learning via SkyRL are also supported.
Generated from reports · updated 13 h ago
Report timeline
Follow the reports to see the event from different sides.
- vLLM BlogPicksvLLM adds day-0 support for Thinking Machines' Inkling 2 Small
vLLM supports Thinking Machines' Inkling 2 Small on day 0, a multimodal text-and-vision MoE with 594B total and 17B active parameters and up to 1M tokens of context. The integration adds kernel fusion with PDL, sequence parallelism for prefill that cuts an 8K-token prefill step by 22% at TP4 and 26% at TP8, and a Gluon relative attention decode kernel that drops a batch-1 decode step at 100K context from 15.1 ms to 4.8 ms. vLLM reports up to 630 TPS/user at TP4 on GB200 with acceptance length 3.5, and also supports LoRA and RL via SkyRL.
Heat for this event
Not enough continuous observations yet to draw a trend.