Skip to content
Hacker News · AI·· 1 d agoAI score60

Standard Kernel uses AI to optimize Intel Gaudi2 past Nvidia H100 on long-context inference

Standard Kernel Used AI to Make Intel's Abandoned Gaudi2 Outperform Nvidia H100

AI summary

Standard Kernel reports that software optimization alone lifted Qwen3.5-27B inference on Intel Gaudi2 to 60% higher decode throughput and 8% higher cached-prefill throughput than an Nvidia H100 80GB PCIe, at 17% of the hourly rental cost. The company says its changes to vLLM and the vllm-gaudi plugin raised Gaudi2's cached-prefill throughput 3.21x and decode throughput 2.84x over baseline, yielding 9.4x more output tokens per dollar. The benchmark used 16 concurrent requests with 70,000-token inputs and a 95% shared prefix.

Source:Hacker News · AI · standardkernel.com