Modal launches Auto Endpoints for self-serve, OpenAI API-compatible LLM inference

Modal launched Auto Endpoints, a self-serve service that deploys production-grade LLM inference as an OpenAI API-compatible endpoint backed by a Modal App the user can see and control. Each supported model ships with a starting deployment tuned from Modal's own benchmarking, using engines like SGLang and the DFlash speculative decoding drafter from Z Lab, and users can inspect GPU choices, engine flags and inference metrics such as TTFT, ITL and speculative decoding acceptance length. Modal also released Modal Servers from beta for regionalized routing with 5ms overhead, and said it generates endpoint deployments with an internal agentic system still monitored by human engineers.