Modal now offers Nvidia B200 and H200 GPUs to any account holder with no sales contact or quota request, priced at $6.25/hour and $4.54/hour respectively. The B200 carries 180 GB of HBM3e memory and 8 TB/s bandwidth, and in Modal's vLLM test running DeepSeek V3 in 8-bit precision, 8xB200 cut median time-to-first-token 2.5x versus 8xH200 at 1 request per second.
DeepSeek V3 processes 47% more tokens per second than R1 at $0.28 per million output tokens, while R1 costs $2.19 per million and runs internal reasoning loops for harder tasks. R1 scores 97.3% on MATH-500 and 79.8% on AIME 2024 versus V3's 90.2% and 39.2%. Most teams route 80-90% of simple queries to V3 and send complex reasoning work to R1.
SRTP secures RTP voice and video streams by adding encrypted payloads, authentication tags, and a master key identifier to each media packet. It relies primarily on AES counter mode and HMAC-SHA1 authentication tags, with sequence numbers blocking replay attacks, and pairs with DTLS-SRTP or SDES for key exchange. SRTP is positioned as a lower-overhead, application-layer option for real-time communications, with GDPR, HIPAA, and CPRA cited as compliance drivers.
A guide on using .env files and environment configuration in voice AI projects, covering dotenv setup for Node.js and Python, adding .env to .gitignore, and Vapi-specific keys like VAPI_API_KEY and VAPI_PROJECT_ID. It cites GitGuardian's finding of 12.8 million secrets leaked on GitHub in 2023, a 28% increase, and recommends separate env files per environment plus tools such as git-crypt, SOPS, AWS Secrets Manager and HashiCorp Vault.
Amazon Lex suits teams already on AWS needing tight integration with Amazon services, while Dialogflow offers broader language support and an easier learning curve for newcomers. Lex uses Alexa's technology with pay-as-you-go pricing including a free tier of 10,000 text and 5,000 speech requests monthly for the first year, and Dialogflow supports over 30 languages with pre-built agents and sentiment analysis.
Vapi's guide explains how to configure environment variables for its Voice AI integration, covering temporary vs. permanent and user-level vs. system-level variables. It details setup on Windows, macOS and Linux with Vapi-specific examples like VAPI_API_KEY and VAPI_MULTILINGUAL_SUPPORT, plus naming and security best practices such as never hardcoding API keys.
Modal launched Modal Batch, a batch-processing interface backed by a rebuilt durable queue that can queue 1 million inputs at once, up from 2,000, and guarantees execution up to 7 days after a job is launched, up from 1 day. A new native handler, .spawn_map, spawns thousands of containers over a set of inputs without users managing async logic, and Harvey reports a 10x speed-up in its document-processing pipeline after fanning inputs across 1,000 containers. Suno uses it to run GPU-accelerated preprocessing on thousands of GPUs, and Achira relies on its retry mechanism for quantum mechanical datasets.
Modal upgraded its serverless Dict key-value store: new Dicts now have unlimited storage instead of the previous 10GiB cap, and item expiry changed to 7 days since last write or read. A new skip_if_exists flag on .put() enables distributed locking and "exactly once" semantics, and Dicts now offer data durability. The changes apply only to newly created Dicts.
Modal built a "resource solver" that uses linear programming and the simplex algorithm to decide which cloud instances to spin up or down, meeting demand for GPUs at predictable prices. The system also captures temporary arbitrage, such as hundreds of H200 GPUs at 20% below the going rate for H100s, saving millions of dollars a year.
Modal released alpha versions of its official SDKs for JavaScript and Go, letting developers call deployed Modal Functions and create isolated sandboxes from Node, Next.js, or Go services without writing Python. The SDKs cannot deploy functions, which still require Python, and only support JSON-like or bytes data types for now. Modal says it can run 10,000 sandboxes in either language without issues.
Lemon Slice built Lemon Slice Live, a product for real-time video conversations with AI characters, using Modal and Daily's low-latency primitives to reach an end-to-end video response latency of 3-6 seconds. The pipeline runs a Pipecat server and a 1B parameter video model on separate Modal containers that communicate via Modal's Tunnel and are co-located with region selection, with Deepgram, Grok and ElevenLabs handling speech-to-text, LLM response and text-to-speech. Earlier, two Python Modal Functions let Lemon Slice scale to 10,000 requests per hour without managing an ECS cluster, job queue or load balancer.
sync. is using Modal to lipsync over 100 hours of video a day, splitting long videos into scenes that run face detection and translation across n T4 GPU containers before passing each scene to its proprietary LipSync GenAI model on A100s. The team deploys up to 95 times a day and shipped 10 major model variants to production in one year, after moving off a Google Colab prototype and AWS Lambda.
Modal now lets each Function queue up to 1 million inputs, up from 2k, with higher .spawn() rate limits and FunctionCall results retained for 7 days. The update also adds ephemeral apps via app.run() inside containers, Git commit info for apps in the CLI and dashboard, and a new TensorRT-LLM example serving LLMs at under 400 ms latency.
Consultant Christian Ulstrup of GSD at Work used Vapi voice AI agents to replace scheduled discovery calls with an asynchronous voice intake system for his AI Action Sessions workshops. Participants call a dedicated Vapi phone line at their convenience, and transcripts plus AI-generated summaries flow into Slack via Zapier and are logged in HubSpot. For one recent client, 20 participants across the globe contributed over 3 hours of discovery, and the session earned a 5 out of 5 rating.
Modal cut GPU, CPU and memory prices by up to 65%, and Team plan users can now run up to 50 GPUs concurrently. The Modal SDK is now on conda-forge starting with version 0.73.46, and a version 1.0 client release is planned. Modal also published an LTX-Video image-to-video example and launched a new logo.
Modal published a guide distinguishing three things commonly called "GPU utilization": GPU Allocation Utilization (GPU-seconds running application code over GPU-seconds paid for), GPU Kernel Utilization (fraction of time running code on GPUs), and Model FLOP/s Utilization (share of theoretical arithmetic bandwidth used). The guide focuses on neural network inference workloads, which Modal calls a revenue center rather than a cost center.
Modal has open sourced its GPU Glossary, an interlinked compendium of short documents on GPU programming topics, on GitHub under a Creative Commons BY 4.0 License. The glossary was first released two months ago to fill a gap in cross-stack GPU reference material, and community requests to contribute prompted the move. The repo's GitHub Issues already list planned extensions, including two sample matmul kernels and a section on Thread Block Clusters.
Modal shipped a refreshed logs dashboard with time-range input filtering, live container profiling for debugging stuck containers or inputs, and Debug Shells that run commands like vim, ps, and strace inside running containers. Region selection is now available on all plans, priced with a multiplier on base prices, and the client adds OIDC authentication for cloud bucket mounts via the new oidc_auth_role_arn field.
Modal introduced memory snapshot restore for its serverless GPU container runtime, which it says makes cold starts about 2.5x faster than standard container startup. A Stable Diffusion inference Function that normally takes around 13 seconds now restores in 3.5 seconds, and a simple import torch program drops from roughly 5 seconds to about 1.05 seconds at p50 and 0.69 seconds at p0. Snapshots are built on gVisor's checkpoint/restore and are created on demand, with automatic fallback to a standard container startup when a restore fails; GPU state is not yet saved, so it must be rebuilt after restore.
Why it matters: Modal explains how gVisor-based memory snapshots cut serverless cold starts, with concrete restore timings for a Stable Diffusion Function and a torch import example.
Modal now offers NVIDIA L40S GPUs at $1.95/hr, with 48GB of memory, up to 40% faster performance on memory-bound tasks and over 100% speedup on compute-bound jobs versus its A10 GPUs. The platform also added Proxy Auth tokens for gating web endpoints, a Filesystem API for Sandboxes, and disk snapshotting that lets users branch new Sandbox instances from snapshots to eliminate cold starts. Modal additionally announced completion of its SOC 2 Type 2 certification.
Modal announced the general availability of Sandboxes, its primitive for safely running untrusted code from LLMs, users, or third-party sources in an isolated environment separate from other Modal resources. Sandboxes offer a simple exec API, runtime-configurable environments via the Image API, and filesystem snapshots for persistence. Modal also upstreamed support into SWE-bench, where a --modal flag lets the Verified benchmark's 500 tasks run in about 7 minutes, and cited Quora's Poe, Codegen, and Relevance AI as customers.
Modal shipped a product update round led by a major upgrade to its beta memory snapshotting feature, which it says can cut cold start times by half for models like Stable Diffusion. The platform also added OpenID Connect support for short-lived identity tokens, eStargz image compression for faster builds from registries like ECR and Docker Hub, and is expanding async job queues toward a 1 million input limit, up from 2k.
Modal now offers NVIDIA L40S GPUs to all users at $1.95/hr, positioned as a faster, larger-memory alternative to its most popular inference accelerator, the A10. The L40S doubles GPU RAM to 48GB, letting models like Flux.1-schnell run in 16bit precision on a single GPU, and delivers roughly 40% speedup on memory-bound jobs and over 100% on compute-bound 16bit Tensor Core jobs.
Fine-tuning adjusts pretrained LLM weights to fit a specific task or domain, delivering higher quality than prompt engineering at lower cost and latency; a fine-tuned Llama 7B can be roughly 50 times more cost-effective per token than GPT-3.5 with comparable performance. The guide covers use cases such as structured output, style adherence and domain-specific instruction, plus options including OpenAI, Predibase, Modal, Google Colab and AWS SageMaker, and frameworks HuggingFace transformers, TRL and Axolotl.
Modal held an internal hackathon where 30 employees formed 13 teams to build projects that all used Modal. Browserman is a multi-modal AI agent built with Llama-3.2-90B-Vision-Instruct-FP8, vLLM and Playwright that navigates the web visually to complete tasks. Glodal prototyped a distributed control plane that cut request latency from 268ms to 3ms, and Modal is now working to build it into the platform.
A top-tier European soccer team, anonymized as AFC Richmond, moved its match tracking data pipeline to Modal's serverless GPU platform and cut the cost of processing a full season of games by 50%. The team runs inference on ~3.5 million tracking observations per game through a custom transformer-based model, and with Airflow on Modal now processes each game's data in minutes rather than hours.
Modal signed a Strategic Collaboration Agreement with AWS to help businesses develop and deploy AI products faster on its serverless platform for GPU-accelerated workloads, which can spin up GPU-enabled containers in as little as one second. As part of the deal, Modal will list on AWS Marketplace and invest in integrations such as AWS PrivateLink to serve customers with specific security and privacy requirements.
OpenArt deployed its proprietary ComfyUI workflows on Modal, cutting deployment from hours on AWS and GCP to minutes and autoscaling to hundreds of GPU containers. Modal now powers 100+ workflows on the platform, which serves over 3 million monthly active users.
Modal shipped static IP proxies that route outbound traffic through workspace-exclusive IPs, plus a native Slack integration for real-time app alerts and a live usage dashboard tracking resources across all workspace apps. The update also adds GPU fallbacks for functions, expanding the pool of resources a function can be scheduled on, and client changes including `modal shell` container IDs and memory snapshotting for parametrized functions.
Modal argues that AI's GPU crunch stems from a mismatch: most demand goes to training, but inference is where revenue comes from, and inference workloads are inherently volatile. Because request rates swing with day-night cycles, weekly patterns, viral traffic and growth trends, fixed long-term GPU reservations push real-world cluster utilization below 50%. Pooling demand across many users — provisioning for the peak of the sum rather than the sum of peaks — can raise utilization sharply, and pooling supply across regions can add further flexibility.
A demo app performs hybrid search over the state of California using timestamps, geolocation data, and satellite image embeddings from the Clay v1 foundation model. Modal orchestrates daily Sentinel API queries, MongoDB Atlas storage, and GPU embedding inference on A10 GPUs, with a small Alpine JS frontend served via FastAPI.
Contextual AI automated and parallelized its GPU-based continuous integration on Modal, replacing manual test runs on in-house GPU nodes. A GitHub Action triggers a Modal Function with multiple GPUs attached, and cached image builds let tests start within 30 seconds of job submission. Modal bills by usage, so 99% of what is billed is actual test run-time.
Modal has added native dynamic batching support, letting inference functions group requests via a single @modal.batched() decorator with max_batch_size and wait_ms parameters. On OpenAI's Whisper large v3 running on an A10G, a max_batch_size of 64 raised throughput from ~1.2 to ~3.3 requests per second per container and cut inference cost by 65%.
Modal shipped app rollbacks that let users revert a deployed app to a previous version from the dashboard or CLI, plus a @batched decorator that accumulates requests into dynamically sized batches to trade latency for throughput and GPU utilization. Sandbox objects can now forward TCP ports to the internet and exec additional processes, and logs can be exported to an OpenTelemetry provider. CPU cores and higher-end GPUs also got cheaper.
Modal now supports HIPAA-compliant workloads, available to Enterprise plan customers who sign a Business Associate Agreement (BAA) via security@modal.com. All Modal features fall within the BAA's scope except Volumes, Images (excluding Filesystem and Directory Snapshots), Memory Snapshots, and user code, where PHI should not be placed. Modal cites its least-privilege data handling and gVisor sandboxing runtime as security foundations, building on its SOC 2 Type 1 announcement and ongoing SOC 2 Type 2 certification.
Cursor has implemented a shadow workspace that lets its AI apply edits in a hidden Electron window and read back lint results without touching the user's editing session. The post outlines the design goals of LSP availability, independence, privacy, concurrency and speed, and explains why copying a TextModel or running separate language servers failed. It also sketches a future kernel-level folder proxy for runnability, noting that macOS and Windows lack a viable FUSE path and that Rust LSP support is currently unavailable.
Modal is cutting prices on CPUs and some of its most powerful GPUs by 15-30%, citing increased competition in the GPU supply pool and the launch of more powerful GPUs over the past year. The company is also changing CPU/memory billing to charge max(reservation, usage) per running container, with a minimum default reservation of 0.125 CPU cores for all Functions. The Team plan rises to $250/mo with unlimited seats and $100/mo in credits, with new pricing taking effect in September for affected Starter and Team/Enterprise users.
Modal's Charles Frye replicated the core claim of the Large Language Monkeys paper on a new dataset, showing that repeated sampling lifts LLaMA 3.1 8B from 66.4% pass@1 to 90.5% pass@100 and 95.1% pass@1000 on HumanEval, versus GPT-4o's reported 90.2% pass@1. The runs used vLLM on Modal serverless GPUs and Modal Sandboxes for parallel test execution, cost well under $50, and showed smooth, predictable scaling across three orders of magnitude.
Why it matters: A hands-on replication shows how repeated sampling with a small open model can approach GPT-4o on HumanEval, offering a concrete, low-cost recipe for trading compute for accuracy.
Modal's monthly product update adds a Datadog integration that forwards application and audit logs with a single click, plus cloud region selection for functions to meet regulatory, egress cost, or database-locality needs. The company also reworked its function call system for higher throughput with significantly lower overhead, targeting low-latency real-time production workloads, and overhauled the app details page with time-interval zoom, container status, and image file system views.
Basis organized the Prompt Olympics, a competitive programming-style prompt engineering contest with a $5,000 prize, and ran it on Modal's infrastructure. In one challenge, participants wrote prompts that steered LLaMA 3 70B, deployed on Modal, to generate code that had to pass tests inside Modal Sandboxes. The finale had players prompt a model to control the Crafter game agent; Basis found even capable open source models like LLaMA 3 70B hard to prompt effectively and switched to a proprietary model API for that component.