Lightreason publishes repo claiming faster Qwen inference on one RTX 4090
Start here
Lightreason published a GitHub repository documenting how it speeds up Qwen inference on a single RTX 4090, reporting 63.0 to 139.6 tok/s for Qwen2.5-3B-Instruct and 7.93 to 51.08 tok/s on a capped Python prompt for Qwen3.6-35B-A3B. Because the 35B text weights occupy 64.56 GiB, the runner keeps most expert weights in system RAM and moves only the ones Qwen uses, while the original BF16 target makes every final token decision. The repository ships no checkpoint or adapter weights. The public 3B Studio quickstart targets Linux with an RTX 4090 and a driver compatible with CUDA 12.8; the 35B runners use a separate benchmark setup requiring more system RAM. Reproducing the published PARD2 numbers requires the matching lab adapter, and the optional Turbo runner and GPU coordination benchmark additionally need the pinned AMD PARD2 draft directory and a locally supplied trained LoRA/projector checkpoint.
Generated from reports · updated 1 h ago
Report timeline
Follow the reports to see the event from different sides.
- Hacker News · AILightreason repo documents 6-11x speedups running Qwen3.6-35B-A3B and Qwen2.5-3B on one RTX 4090
Lightreason published a GitHub repo with its process for speeding up Qwen inference on a single RTX 4090, reporting 63.0 to 139.6 tok/s for Qwen2.5-3B-Instruct and 7.93 to 51.08 tok/s on a capped Python prompt for Qwen3.6-35B-A3B. The 35B text weights occupy 64.56 GiB, so the runner keeps most expert weights in system RAM and moves only the ones Qwen uses, while the original BF16 target makes every final token decision. The checkout ships no checkpoint or adapter weights, and the 3B Studio quickstart targets Linux with CUDA 12.8 and PyTorch 2.9.1.
Heat for this event
Not enough continuous observations yet to draw a trend.