How Modal achieved truly serverless GPUs

Modal published an engineering account of how it cut AI inference replica spin-up from roughly 2,000 seconds to about 50 seconds using four components: a buffer of idle healthy GPUs, a lazy content-addressed filesystem called ImageFS, host-side checkpoint/restore via gVisor's runsc, and CUDA checkpoint/restore built on Nvidia driver support. Across over ten thousand cold starts, mean vLLM boot latency fell from 95,679 ms to 13,797 ms and SGLang from 83,713 ms to 17,486 ms for a ~1 GiB Qwen 3 0.6B model in bf16. The post also notes limits, including single-GPU-only snapshots and host-environment sensitivity that requires multiple snapshots per deployment.