vLLM Blog· vLLM Team, Red Hat PyTorch Team, and Meta PyTorch Team·· 8 d agoAI score58
vLLM migrates its CUDA build to the LibTorch Stable ABI
Migrating vLLM to the LibTorch Stable ABI
AI summary
vLLM has migrated its CUDA build to the LibTorch Stable ABI, and as of v0.33.0 it publishes LibTorch ABI Stable CUDA wheels, so a new PyTorch wheel can be imported without rebuilding vLLM. The team moved in-tree sources and seven LibTorch-dependent dependencies, including FlashMLA, DeepGEMM and FlashAttention-2/3, onto stable APIs, using torch-abi-audit in CI to confirm no unstable symbols remain. The post notes tradeoffs such as a limited stable API subset and small host-side overhead, and says CPU and ROCm builds are next.
Source:vLLM Blog · vllm.ai