Modal profiled SGLang's scheduler on a multimodal workload and replaced repeated CUDA IPC handle book-keeping with a simple Python dict cache, lifting throughput 16.2% and cutting mean TTFT 13.2% and TPOT 17.2% on Qwen2.5-VL-3B-Instruct on H100s. The fix is merged in SGLang v0.5.10.
AE Studio details how it trained LLMs to prove math theorems with reinforcement learning on Modal, comparing Evolution Strategies against GRPO while keeping the training loop fixed. The setup runs vLLM inference on GPUs, Lean compiler verification in isolated CPU sandboxes, and a lightweight orchestrator, using per-function images, .map() fan-out, and Volumes for base weights.
GRASP is a new gradient-based planner for learned world models that makes long-horizon planning practical by lifting trajectories into virtual states for parallel-in-time optimization, adding stochasticity to state iterates for exploration, and reshaping gradients so actions receive clean signals while avoiding brittle state-input gradients through high-dimensional vision models. The work addresses ill-conditioned computation graphs, non-greedy local minima, and high-dimensional latent failure modes that make long-horizon planning fragile. It was done with Mike Rabbat, Aditi Krishnapriyan, Yann LeCun, and Amir Bar.
Berkeley AI Research researchers describe SPEX and ProxySPEX, algorithms that identify influential feature, data, and model-component interactions in LLMs at scales far beyond prior methods. SPEX reframes interaction search as sparse recovery using sparsity and low-degreeness, while ProxySPEX exploits hierarchy to match SPEX with about 10x fewer ablations. On sentiment analysis, SPEX matches Faith-Shap and Faith-Banzhaf faithfulness on short inputs and retains it as context scales to thousands of features.
Continual learning is inseparable from specialization because the cost of remembering scales with scope: as context grows, the KV cache grows linearly and can exceed all model weights, so information must move from token level into weights. Task specialization is essentially solved via continuous fine-tuning pipelines, while broad-scope, sparse-feedback, one-shot retention remains the hard problem.
Reducto migrated inference workloads for its 30+ document-processing models to Modal, cutting P90 latency by 3x. GPU memory snapshotting reduced cold boots by 83%, from ~70s to ~12s, while per-customer autoscaling pools isolate traffic spikes. In a load test, Reducto scaled its ingestion pipeline to over 1,000 GPUs in under an hour to demonstrate 100k pages per minute.
Modal's blog explains how host overhead—gaps where the GPU sits idle waiting for the CPU to prepare work—shows up as low GPU kernel utilization in AI inference. At 50% host overhead, GPU inference costs are 2x what they could be. The post details fixes used in production open source inference engines, including constructing tensors on-device instead of transferring them and reducing unnecessary CUDA kernel launches.
Decagon and Modal partnered to build Decagon Voice 2.0, cutting latency by 65% while improving intent recognition and response quality. Decagon fine-tuned compact open-source models with SFT and RL to surpass larger frontier models, while Modal trained a custom EAGLE3 draft model with 38% higher accept lengths and re-engineered SGLang with an async speculative decoding scheduler, boosting throughput up to 12%.
Modal engineers read the released Flash Attention 4 source code and reverse-engineered how the Blackwell-optimized attention kernel works, since no official technical report exists yet. The write-up traces a tile through the kernel's warp-specialized asynchronous pipeline and details two math tricks: a cubic-polynomial software exponential that can replace SFU-based exponentiation, and a smarter online softmax rescaling rule that Tri Dao said cuts corrections by a factor of 10. FA4 is reported to be roughly 20% faster than Nvidia's closed-source cudnn attention kernels, and a backwards version was not yet available at the time of writing.
Why it matters: Modal engineers read the released Flash Attention 4 source and explain its warp-specialized pipeline and two math tricks, offering a transferable view of where CUDA kernel design is heading.
Modal announced GPU memory snapshots in alpha, extending its existing checkpoint/restore system to capture GPU vRAM, CUDA kernels, streams and contexts so GPU workloads restore from a frozen state instead of rebuilding CUDA state after boot.
Why it matters: Modal adds GPU state to its memory snapshots, cutting cold boots for torch.compile and vLLM workloads by up to 10x and making scale-to-zero more practical.
A Modal Blog post argues that "dollars per token" pricing, inherited from API providers like OpenAI, Anthropic and Alphabet, is actively harmful for teams running self-hosted LLM inference. It contends that application users care about requests rather than tokens, and that per-request framing is needed to reason about latency, replica counts and compute costs.
Modal has launched multi-node training clusters in beta, letting users tap dozens of GPUs across multiple hosts by adding the @clustered decorator, with 3.2 Tbps Infiniband RDMA interconnect for near-linear scaling. The platform now offers NVIDIA B200 and H200 GPUs serverlessly at $6.25/hr and $4.54/hr respectively, claiming up to 2-4× faster LLM inference than H100s. Modal also released Client 1.0, emphasizing API stability, alongside updates including a read-only Volume method and timezone support for Cron schedules.
Mistral and Llama 3 split on philosophy for voice AI: Mistral's 7B to Mistral Small 3.1 (24B) models favor low-latency efficiency, while Llama 3 scales from 8B to 70B for stronger reasoning and multilingual reach. Mistral Small 3.1 reaches 0.29s time to first token at 150-166 tokens per second, and Mistral Small 24B beats Llama 3.1 8B Instruct on ARC-C, GPQA and MMLU, while Llama 3.1 405B scores 85.2% on MMLU and 96.8% on GSM8K.
Google's Gemini 2.5 Flash and 2.5 Pro differ mainly in speed versus reasoning depth, both offering 1M-token context windows and native availability in Vapi for voice agents. Flash returns its first token in 0.21–0.37 seconds at 163 tokens per second and costs $0.15/$0.60 per million input/output tokens, while Pro costs $1.25/$10 and adds a Deep Think mode, with a 2M-token window planned.
Modal now offers Nvidia B200 and H200 GPUs to any account holder with no sales contact or quota request, priced at $6.25/hour and $4.54/hour respectively. The B200 carries 180 GB of HBM3e memory and 8 TB/s bandwidth, and in Modal's vLLM test running DeepSeek V3 in 8-bit precision, 8xB200 cut median time-to-first-token 2.5x versus 8xH200 at 1 request per second.
DeepSeek V3 processes 47% more tokens per second than R1 at $0.28 per million output tokens, while R1 costs $2.19 per million and runs internal reasoning loops for harder tasks. R1 scores 97.3% on MATH-500 and 79.8% on AIME 2024 versus V3's 90.2% and 39.2%. Most teams route 80-90% of simple queries to V3 and send complex reasoning work to R1.
Modal published a guide distinguishing three things commonly called "GPU utilization": GPU Allocation Utilization (GPU-seconds running application code over GPU-seconds paid for), GPU Kernel Utilization (fraction of time running code on GPUs), and Model FLOP/s Utilization (share of theoretical arithmetic bandwidth used). The guide focuses on neural network inference workloads, which Modal calls a revenue center rather than a cost center.
Modal introduced memory snapshot restore for its serverless GPU container runtime, which it says makes cold starts about 2.5x faster than standard container startup. A Stable Diffusion inference Function that normally takes around 13 seconds now restores in 3.5 seconds, and a simple import torch program drops from roughly 5 seconds to about 1.05 seconds at p50 and 0.69 seconds at p0. Snapshots are built on gVisor's checkpoint/restore and are created on demand, with automatic fallback to a standard container startup when a restore fails; GPU state is not yet saved, so it must be rebuilt after restore.
Why it matters: Modal explains how gVisor-based memory snapshots cut serverless cold starts, with concrete restore timings for a Stable Diffusion Function and a torch import example.
Modal now offers NVIDIA L40S GPUs to all users at $1.95/hr, positioned as a faster, larger-memory alternative to its most popular inference accelerator, the A10. The L40S doubles GPU RAM to 48GB, letting models like Flux.1-schnell run in 16bit precision on a single GPU, and delivers roughly 40% speedup on memory-bound jobs and over 100% on compute-bound 16bit Tensor Core jobs.
Modal has added native dynamic batching support, letting inference functions group requests via a single @modal.batched() decorator with max_batch_size and wait_ms parameters. On OpenAI's Whisper large v3 running on an A10G, a max_batch_size of 64 raised throughput from ~1.2 to ~3.3 requests per second per container and cut inference cost by 65%.
Modal's Charles Frye replicated the core claim of the Large Language Monkeys paper on a new dataset, showing that repeated sampling lifts LLaMA 3.1 8B from 66.4% pass@1 to 90.5% pass@100 and 95.1% pass@1000 on HumanEval, versus GPT-4o's reported 90.2% pass@1. The runs used vLLM on Modal serverless GPUs and Modal Sandboxes for parallel test execution, cost well under $50, and showed smooth, predictable scaling across three orders of magnitude.
Why it matters: A hands-on replication shows how repeated sampling with a small open model can approach GPT-4o on HumanEval, offering a concrete, low-cost recipe for trading compute for accuracy.
Cursor published a follow-up to its original "Problems" post, listing the research questions it considers most critical for the next phase of AI coding. The company has extended Copilot++ to predict the next cursor position, and combining that with next-edit prediction lets the model "play" sequences of low-entropy changes—a demo required 11 Tab presses versus 3 presses of every other key, which Cursor calls Cursor Flow. Other directions include action prediction across the codebase, perfect edits via more inference-time compute, hallucinated pseudocode, multi-file editing, optimal context, multi-hop context, and bug detection and debugging.
Cursor measured Llama-2-70B inference on 2 80-GB A100 GPUs and found it over 3x cheaper than GPT-3.5 for prompt tokens but far more expensive for generated tokens at small batch sizes. Prompt processing costs about $0.00042 per 1K tokens versus GPT-3.5's $0.0015, while generation reaches $0.066 per 1K tokens at batch size 1 and only approaches GPT-3.5 pricing at large batches that push time to first token to several seconds. Cursor recommends open-source models for prompt-heavy tasks like classification and reranking, and closed-source APIs for generation-heavy workloads.
Why it matters: Cursor's measured cost and latency data shows when self-hosting Llama-2-70B beats GPT-3.5 and when it does not, offering a reusable framework for open-source model serving decisions.