Transformers adds MoE expert streaming, runs 180B model on 32GB Mac
Start here
A developer added SSD-based expert streaming to Hugging Face Transformers via PR #50717, allowing a 180B-class mixture-of-experts model to run on a 32 GB MacBook M5. The demo used Qwen3.8-Flash-Next with 121 GB of FP8 experts, decoding at 13.9 tok/s — about 6% behind Strata's 14.8 tok/s. The method keeps only routed experts in a bounded device slot pool so grouped_mm runs unchanged, quantizes experts to 3 bits (49 GB on disk, 12 GiB cached), and leaves the 51 GB n-gram table on disk. Performance is uneven: on a single A10G with Qwen3.6-35B-A3B, Transformers + Accelerate reached 0.16 tok/s, versus 82–88 tok/s for Strata and 63–64 tok/s for llama.cpp.
Generated from reports · updated 45 min ago
Report timeline
Follow the reports to see the event from different sides.
- Hugging Face BlogPicksTransformers adds MoE expert streaming, running a 180B model on a 32GB Mac within 6% of Strata
A developer added SSD-based expert streaming to Transformers via PR #50717, running Qwen3.8-Flash-Next with 121 GB of FP8 experts on a 32 GB MacBook M5 at 13.9 tok/s decode, about 6% behind Strata's 14.8 tok/s. The approach keeps only routed experts in a bounded device slot pool so grouped_mm runs unchanged, quantizes experts to 3 bits (49 GB on disk, 12 GiB cached), and keeps the 51 GB n-gram table on disk. Benchmarks on one A10G with Qwen3.6-35B-A3B put Transformers + Accelerate at 0.16 tok/s versus 82-88 for Strata and 63-64 for llama.cpp.
Heat for this event
Not enough continuous observations yet to draw a trend.