InferSSD runs 800B-class LLMs on consumer hardware by keeping expert weights on NVMe

InferSSD, an open-source inference runtime built on llama.cpp, runs 763B-class DeepSeek-V4.1-Flash-1017 and 816B-class Muse-Spark-Open on consumer machines by keeping routed expert weights on NVMe storage while attention, shared layers and the KV cache stay in RAM or VRAM. On an RTX 5090 with 32 GB RAM and 32 GB VRAM it reports 164.5 tokens/s prefill and 7.27 tokens/s decode for DeepSeek at a 4K prompt tier, versus 8.06 and 0.867 tokens/s for llama.cpp on the same system. The article also estimates that heavy daily use could mean roughly 85 TB of host reads per day and lays out a conditional SSD wear model, noting the figures are assumptions to test rather than a promised drive lifespan.