InferSSD runs 800B-class LLMs on consumer hardware via NVMe
Start here
WiCi released InferSSD, an open-source runtime built on llama.cpp that runs 800B-class models on consumer machines by keeping routed expert weights on NVMe storage while attention, shared layers and the KV cache stay in RAM or VRAM. It is available under Apache-2.0; users download model weights separately, keep all GGUF shards together, and the first launch profiles the machine and drive, which the guide says takes one to three minutes. Model weights remain subject to their publishers' licenses. On an RTX 5090 with 32 GB RAM and 32 GB VRAM, InferSSD reports 164.5 tokens/s prefill and 7.27 tokens/s decode for 763B-class DeepSeek-V4.1-Flash-1017 at a 4K prompt tier, versus 8.06 and 0.867 tokens/s for llama.cpp on the same system; it also runs 816B-class Muse-Spark-Open. The article estimates heavy daily use could mean roughly 85 TB of host reads per day and presents a conditional SSD wear model, which it describes as assumptions to test rather than a promised drive lifespan.
Generated from reports · updated 12 h ago
Report timeline
Follow the reports to see the event from different sides.
- Hacker News · AIInferSSD runs 800B-class LLMs on consumer hardware by keeping expert weights on NVMe
InferSSD, an open-source inference runtime built on llama.cpp, runs 763B-class DeepSeek-V4.1-Flash-1017 and 816B-class Muse-Spark-Open on consumer machines by keeping routed expert weights on NVMe storage while attention, shared layers and the KV cache stay in RAM or VRAM. On an RTX 5090 with 32 GB RAM and 32 GB VRAM it reports 164.5 tokens/s prefill and 7.27 tokens/s decode for DeepSeek at a 4K prompt tier, versus 8.06 and 0.867 tokens/s for llama.cpp on the same system. The article also estimates that heavy daily use could mean roughly 85 TB of host reads per day and lays out a conditional SSD wear model, noting the figures are assumptions to test rather than a promised drive lifespan.
Heat for this event
Not enough continuous observations yet to draw a trend.