Transformers adds MoE expert streaming, running a 180B model on a 32GB Mac within 6% of Strata
180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers
A developer added SSD-based expert streaming to Transformers via PR #50717, running Qwen3.8-Flash-Next with 121 GB of FP8 experts on a 32 GB MacBook M5 at 13.9 tok/s decode, about 6% behind Strata's 14.8 tok/s. The approach keeps only routed experts in a bounded device slot pool so grouped_mm runs unchanged, quantizes experts to 3 bits (49 GB on disk, 12 GiB cached), and keeps the 51 GB n-gram table on disk. Benchmarks on one A10G with Qwen3.6-35B-A3B put Transformers + Accelerate at 0.16 tok/s versus 82-88 for Strata and 63-64 for llama.cpp.
A hands-on account of adding SSD expert streaming to Transformers, showing how a 180B MoE model runs on a 32GB Mac and where the remaining host-sync overhead sits.
Source:Hugging Face Blog · huggingface.co