Hacker News · AI·· 2 h agoAI score58
llama.cpp v1.0 benchmarked on an RTX 3090: up to 55% faster with speculative decoding
Llama.cpp v1.0 on an RTX 3090: up to 55% faster with speculative decoding
AI summary
A hands-on test of llama.cpp v1.0.0 (build 12359) against a pre-v1 build on an RTX 3090 found single-request speed unchanged within 2%, while four parallel requests produced 14% more text per second with the 9B model and 22% more with the 27B model. Speculative decoding with MTP heads made the 27B model 52 to 55% faster, and the gains require an NVIDIA RTX 30-series card or newer. The update removes the --defrag-thold flag and requires an API key for tools or MCP servers on non-loopback addresses, and Ollama 0.40.16 still ships an older llama.cpp build.
Source:Hacker News · AI · homebrewserver.club