vLLM adds day-0 support for Thinking Machines' Inkling 2 Small

vLLM added day-0 inference support for Thinking Machines' Inkling 2 Small, a multimodal text-and-vision mixture-of-experts model with 594B total and 17B active parameters and up to 1M tokens of context. vLLM says the model can be served on a single node. The integration adds kernel fusion with PDL, sequence parallelism for prefill that vLLM says cuts an 8K-token prefill step by 22% at TP4 and 26% at TP8, and a Gluon relative attention decode kernel that reduces a batch-1 decode step at 100K context from 15.1 ms to 4.8 ms. vLLM reports up to 630 TPS/user at TP4 on GB200 with acceptance length 3.5, and says LoRA and reinforcement learning via SkyRL are also supported.

สร้างจากรายงาน · อัปเดต 14 ชม. ที่แล้ว

ยังมีการเฝ้าดูต่อเนื่องไม่พอจะวาดแนวโน้ม