InferSSD runs 800B-class LLMs on consumer hardware via NVMe

WiCi released InferSSD, an open-source runtime built on llama.cpp that runs 800B-class models on consumer machines by keeping routed expert weights on NVMe storage while attention, shared layers and the KV cache stay in RAM or VRAM. It is available under Apache-2.0; users download model weights separately, keep all GGUF shards together, and the first launch profiles the machine and drive, which the guide says takes one to three minutes. Model weights remain subject to their publishers' licenses. On an RTX 5090 with 32 GB RAM and 32 GB VRAM, InferSSD reports 164.5 tokens/s prefill and 7.27 tokens/s decode for 763B-class DeepSeek-V4.1-Flash-1017 at a 4K prompt tier, versus 8.06 and 0.867 tokens/s for llama.cpp on the same system; it also runs 816B-class Muse-Spark-Open. The article estimates heavy daily use could mean roughly 85 TB of host reads per day and presents a conditional SSD wear model, which it describes as assumptions to test rather than a promised drive lifespan.

สร้างจากรายงาน · อัปเดต 12 ชม. ที่แล้ว

ยังมีการเฝ้าดูต่อเนื่องไม่พอจะวาดแนวโน้ม