Skip to content
#Deployment / engineering
January 15Thursday
  1. Modal Blog22

    Chai Discovery scales computational drug discovery on Modal

    Chai Discovery is using Modal to run its machine learning drug discovery pipelines, from protein embeddings and multiple sequence alignments to antibody design. Modal Volumes downloads and indexes datasets reaching hundreds of gigabytes once, then shares them across machines, while elastic GPU scaling spins up hundreds of GPUs in minutes. Chai says the same setup now carries research workflows into production without rewriting infrastructure.

January 6Tuesday
  1. OpenAI Developers Blog42

    OpenAI Launches Docs MCP Server for Developer Documentation

    OpenAI released a public Model Context Protocol (MCP) server that gives agents read-only access to developer documentation across developers.openai.com, platform.openai.com, and learn.chatgpt.com. The server is documentation-only and does not call the OpenAI API on the user's behalf. Developers can add it to Codex via the CLI or IDE extension with a single shared configuration, and pair it with the OpenAI Docs Skill for citation-backed answers.

December 28, 2025Sunday
  1. Modal Blog38

    Modal on Keeping 20,000+ GPUs Healthy Across Clouds

    Modal has scaled its multi-cloud GPU worker pool past 20,000 concurrent GPUs and launched over four million cloud instances in the last couple years, and is now sharing the reliability system behind it. The post covers instance type testing and selection, machine image preparation and boot checks, passive and active GPU healthchecking, and observability and support. It reports hyperscaler differences such as Cloud A's 99.6% boot success and Cloud C's H100s running above 90ºC in 2025.

December 17, 2025Wednesday
  1. Vapi Blog38

    Vapi Voices Beta Launches at $0.0025/min for High-Volume Voice AI

    Vapi released Vapi Voices Beta, a family of voice models built for high-volume, cost-sensitive voice AI use cases, priced at $0.0025 per minute with latency optimized for conversational responsiveness. The models appear in the existing voice dropdown alongside providers like ElevenLabs, PlayHT and Deepgram, so teams can route high-volume, low-stakes flows such as appointment reminders and routing menus to them with zero integration changes. Vapi says it is gating general availability on what beta usage reveals about use cases, latency under real load and whether the savings matter.

December 2, 2025Tuesday
  1. Modal Blog42

    Modal adds Day 0 support for Mistral 3 with GPU snapshotting for 10x faster cold starts

    Modal now offers Day 0 support for Mistral 3, letting developers deploy the open model family on serverless GPUs without managing infrastructure. On the 3B Ministral 3 model, Modal's GPU snapshotting feature cut median cold start time from about 118s to about 12s, nearly a 10x reduction. Ministral 3 comes in 3B, 8B, and 14B sizes and performs competitively with the Qwen 3-VL series on benchmarks.

November 20, 2025Thursday
  1. Modal Blog22

    Modal on why coding agents need good developer experience too

    Modal argues that the same developer-experience principles that help humans—tight feedback loops, programmatic access via CLI/SDK/API, actionable error messages, and consistent code-adjacent examples—also make coding agents faster. Because agents iterate by generating, running, and observing code, vague errors or UI-only config stall them, so designing for agents means applying the same principles rather than reinventing them.

  2. Vapi Blog42

    Vapi adds structured outputs to extract schema-validated data from voice AI calls

    Vapi now supports structured outputs for all voice AI assistants, ensuring every call returns data that matches a developer-defined JSON schema. Schemas are enforced at runtime using provider-native capabilities from OpenAI, Anthropic, Gemini and others, and can be reused across multiple assistants. Extracted data arrives in call completion webhooks and the REST API.

  3. Replit Blog42

    Replit launches Fast Mode for quicker, cheaper Agent edits

    Replit launched Fast Mode, a faster way to direct its Agent to make small, precise changes. It runs on an efficient agentic model that is significantly more affordable than Build mode and optimized to finish requests in the fewest steps, aimed at quick fixes, visual iterations and small features. Fast Mode is available to all Replit users starting today and is activated via the lightning bolt icon.

November 19, 2025Wednesday
  1. Modal Blog30

    How Reducto Cut Enterprise Document Processing P90 Latency 3x on Modal

    Reducto migrated inference workloads for its 30+ document-processing models to Modal, cutting P90 latency by 3x. GPU memory snapshotting reduced cold boots by 83%, from ~70s to ~12s, while per-customer autoscaling pools isolate traffic spikes. In a load test, Reducto scaled its ingestion pipeline to over 1,000 GPUs in under an hour to demonstrate 100k pages per minute.

November 18, 2025Tuesday
  1. Modal Blog36

    Modal Blog: Host Overhead Is Killing Your Inference Efficiency

    Modal's blog explains how host overhead—gaps where the GPU sits idle waiting for the CPU to prepare work—shows up as low GPU kernel utilization in AI inference. At 50% host overhead, GPU inference costs are 2x what they could be. The post details fixes used in production open source inference engines, including constructing tensors on-device instead of transferring them and reducing unnecessary CUDA kernel launches.

November 13, 2025Thursday
  1. Modal Blog40

    How Decagon Shipped Real-Time Voice AI on Modal

    Decagon and Modal partnered to build Decagon Voice 2.0, cutting latency by 65% while improving intent recognition and response quality. Decagon fine-tuned compact open-source models with SFT and RL to surpass larger frontier models, while Modal trained a custom EAGLE3 draft model with 38% higher accept lengths and re-engineered SGLang with an async speculative decoding scheduler, boosting throughput up to 12%.

November 4, 2025Tuesday
  1. Modal Blog58

    Modal, Pipecat and open models achieve one-second voice-to-voice latency

    Modal's Ben Shababo walks through building a conversational voice bot on Modal with the open source Pipecat framework and open weights models, reaching a median voice-to-voice latency of about one second when the client and containers are geographically close. The pipeline uses NVIDIA parakeet-tdt-0.6b-v3 for STT, Qwen3-4B-Instruct-2507 served with vLLM, KokoroTTS for speech output, and ChromaDB with all-minilm-l6-v2 for RAG over Modal's docs. Modal Tunnels bypass the input plane for low-latency WebSocket and HTTP links to the STT, TTS and vLLM services, while a FunctionCall tied to each conversation session recovers autoscaling, and pinning services to nearby regions cuts network latency.

October 31, 2025Friday
  1. Modal Blog22

    Modal updates Volumes v2, JavaScript and Go SDKs to beta

    Modal has released Volumes v2 in open beta for all apps, with a filesystem rebuilt for higher throughput, faster random access, and true high-concurrency writes from hundreds of containers, plus no cap on stored files. Its JavaScript/TypeScript and Go SDKs also entered beta at v0.5, adding a unified Client object and support for Sandboxes, Functions, Images, and Volumes.

October 30, 2025Thursday
  1. Modal Blog38

    Modal and Datalab: Deploy Marker and Surya Document Intelligence in Under 5 Minutes

    Modal has partnered with Datalab to let developers deploy the Marker document-parsing model and Surya OCR toolkit on Modal GPUs in under five minutes. Marker is a sub-billion-parameter model supporting over 90 languages, and running it on Modal reaches about 2.2 pages per second per container, roughly 10x the 0.22 pages per second on an M4 Mac using Apple MPS. Datalab's tools stay free for research, personal use, and startups under $2M in funding or revenue, with commercial licensing available.

October 27, 2025Monday
  1. Vapi Blog28

    Vapi Offers a Free Week of Cartesia Sonic-3 TTS

    Vapi is giving its community free, unrestricted access to Cartesia's newly launched Sonic-3 TTS model through its managed API key from October 28 through November 2. Sonic-3 adds emotion and laughter tags, fine-grained speed and volume controls, and support for 27 new languages. Vapi recommends starting with the Kira and Ariana voices for low-latency, natural-sounding agents.

September 26, 2025Friday
  1. Modal Blog60

    Modal engineers reverse-engineer how Flash Attention 4 works

    Modal engineers read the released Flash Attention 4 source code and reverse-engineered how the Blackwell-optimized attention kernel works, since no official technical report exists yet. The write-up traces a tile through the kernel's warp-specialized asynchronous pipeline and details two math tricks: a cubic-polynomial software exponential that can replace SFU-based exponentiation, and a smarter online softmax rescaling rule that Tri Dao said cuts corrections by a factor of 10. FA4 is reported to be roughly 20% faster than Nvidia's closed-source cudnn attention kernels, and a backwards version was not yet available at the time of writing.

    Why it matters: Modal engineers read the released Flash Attention 4 source and explain its warp-specialized pipeline and two math tricks, offering a transferable view of where CUDA kernel design is heading.

September 22, 2025Monday
  1. Modal Blog38

    Modal Vibe: An AI Coding Platform Built on Modal Sandboxes That Scales to 1,000 Apps in 5 Minutes

    Modal built Modal Vibe, an open-source demo vibe coding platform on Modal Sandboxes that scales from 0 to 1,000 AI-coded apps in about five minutes. A load test creating 1,000 apps, each in its own Sandbox, took roughly 300 seconds, with the AI API as the rate-limiting step. Modal says the Team Plan can run 1,000 concurrent Sandboxes, enough for over one million sessions per month.

September 19, 2025Friday
  1. Modal Blog20

    Modal Product Updates: Notebooks, Sandbox Idle Timeouts, and Client Changes

    Modal shipped product updates including Modal Notebooks for cloud-based Python experimentation with up to 8 B200 GPUs, plus an optional idle_timeout parameter for modal.Sandbox.create(). The client also added a startup_timeout parameter to @app.function() and @app.cls() decorators and a new API pattern for imperative management of Modal resource types. Modal Notebooks have since been shut down, though existing notebooks can still be downloaded as .ipynb files.

September 16, 2025Tuesday
  1. Modal Blog47

    Inside Modal Notebooks: How Modal Built a Cloud GPU Notebook That Boots in Seconds

    Modal detailed the engineering behind Modal Notebooks, its cloud Jupyter notebook that boots GPUs and custom images in seconds with real-time collaboration. Kernels run inside Modal Sandboxes, with a modal-kernelshim daemon translating Jupyter ZeroMQ messages into HTTP calls tunneled through Modal's control plane. A lazy-loading Rust FUSE container filesystem fetches file contents on demand through a tiered cache, avoiding the image-unpacking latency that can approach a minute for an 8 GB Python/ML image.

September 9, 2025Tuesday
  1. Modal Blog58

    Modal launches Notebooks with GPU kernels that start in under 5 seconds

    Modal Notebooks is now generally available to all Modal customers after a limited beta last month. The collaborative notebook environment runs on a GPU-enabled Python kernel that reaches a ready state in under 5 seconds on arbitrary container images and hardware up to 256 vCPUs and 8 H100/B200 GPUs, with auto-idle shutdown, shared Volumes and Secrets, and real-time multi-cursor editing. Modal says over 5,000 accounts ran 200,000 code cells during the August beta, and upcoming features include memory snapshots, one-click export to Modal Apps, and scheduled runs.

August 28, 2025Thursday
  1. Modal Blog22

    How Zencastr transcribed hundreds of years of audio in days with Modal

    Zencastr moved its podcast transcription workloads to Modal's GPU platform, scaling to 1,500 concurrent GPUs within minutes to process hundreds of years of audio in just a few days. The shift from self-managed Kubernetes GPU nodes to Modal's scale-to-zero infrastructure cut idle GPU costs and removed driver and dependency management overhead. All core ML workloads now run on Modal, including transcription, speaker counting, laughter detection, and audio quality scoring.

August 11, 2025Monday
  1. Modal Blog38

    Modal Product Updates: GPU Memory Snapshots, Notebooks, Service Tokens, and More

    Modal added GPU memory snapshots in alpha, cutting cold boots for GPU workloads by 10x by skipping steps like Torch compilation, enabled via the enable_gpu_snapshot option. The update also brings Modal Notebooks in beta for running Python in the browser, Service Tokens for workspace managers, and browser-generated UIs for deployed Functions.

July 30, 2025Wednesday
  1. Modal Blog60

    Modal launches GPU memory snapshots for sub-second serverless cold starts

    Modal announced GPU memory snapshots in alpha, extending its existing checkpoint/restore system to capture GPU vRAM, CUDA kernels, streams and contexts so GPU workloads restore from a frozen state instead of rebuilding CUDA state after boot.

    Why it matters: Modal adds GPU state to its memory snapshots, cutting cold boots for torch.compile and vLLM workloads by up to 10x and making scale-to-zero more practical.

July 24, 2025Thursday
  1. Modal Blog36

    What is an AI code sandbox?

    An AI code sandbox is an isolated environment that executes LLM-generated code at runtime, containing the blast radius of errors or injected vulnerabilities. Modern implementations run on VMs or containers, often using secure runtimes like gVisor to intercept syscalls and shield the host. Most companies choose managed sandbox services over building their own with Kubernetes, since cold starts and scaling to millions of concurrent sessions are costly to handle.

July 23, 2025Wednesday
  1. Modal Blog52

    Modal benchmarks open ASR models for batch transcription at 100x speed and cost gains

    Modal implemented a batch transcription service on its platform using NVIDIA's parakeet-tdt-0.6b-v2 and canary-1b-flash, reporting over 100x faster or 100x cheaper end-to-end performance than a popular proprietary ASR API while matching its error rate on roughly a week of ESB benchmark audio. The team attributes the gains to shuffling files across request batches, sorting audio by duration within GPU batches, parallel downloads, and empirically chosen GPU and worker configurations. The post walks through the architecture and optimization steps so others can build similar services.

July 16, 2025Wednesday
  1. Modal Blog30

    Dollars per token considered harmful: why self-hosted LLM inference should be priced per request

    A Modal Blog post argues that "dollars per token" pricing, inherited from API providers like OpenAI, Anthropic and Alphabet, is actively harmful for teams running self-hosted LLM inference. It contends that application users care about requests rather than tokens, and that per-request framing is needed to reason about latency, replica counts and compute costs.

July 11, 2025Friday
  1. Modal Blog40

    Modal Product Updates: Multi-node Training Clusters, B200 and H200 GPUs, and Client 1.0

    Modal has launched multi-node training clusters in beta, letting users tap dozens of GPUs across multiple hosts by adding the @clustered decorator, with 3.2 Tbps Infiniband RDMA interconnect for near-linear scaling. The platform now offers NVIDIA B200 and H200 GPUs serverlessly at $6.25/hr and $4.54/hr respectively, claiming up to 2-4× faster LLM inference than H100s. Modal also released Client 1.0, emphasizing API stability, alongside updates including a read-only Volume method and timezone support for Cron schedules.

  2. Modal Blog31

    Jamsocket is joining Modal

    Jamsocket, the backend platform behind the open-source orchestrator Plane, Y-Sweet and ForeverVM, is joining Modal, with co-founders Paul Butler and Taylor Baldwin joining the Modal team. Jamsocket products will keep operating normally while core functionality is migrated to Modal, which the founders say lets AI-centric customers deploy GPU inference, code sandboxes and real-time backends from one platform.

July 7, 2025Monday
  1. Modal Blog28

    How Modal Powered 250,000 Lovable App Creations in a Weekend

    Lovable used Modal Sandboxes to run LLM-generated code during a June 2025 promotional weekend with Anthropic, OpenAI, and Google, enabling users to build an estimated 250,000 applications in 48 hours. Modal ran over 1 million sandboxes with up to 20,000 concurrent at peak as Lovable's concurrent sessions surged 2.5-3x, and its platform team was not paged once. Lovable cut its sandbox orchestration code from 15,000 lines to 700 lines after switching to Modal.

July 2, 2025Wednesday
  1. Modal Blog40

    How Modal Used Evals and Inference-Time Compute Scaling to Generate Scannable QArt Codes

    Modal used evals and inference-time compute scaling to improve its QArt code system, boosting QR scan rate to its 95% service-level objective while improving aesthetic quality and returning codes in under 20s at p95. The work follows the standard ML playbook of collecting human judgments, aligning to them, then scaling, addressing the "it worked on my prompt" gap that leaves many generated QR codes unscannable.

June 30, 2025Monday
  1. Modal Blog28

    How Quora uses Modal to run thousands of Python sandboxes simultaneously

    Quora uses Modal Sandboxes to securely execute LLM-generated code in Poe, its AI chatbot platform, shipping the feature months earlier than an in-house build would have allowed. The team stress-tested Sandbox creation throughput to 1000 sandboxes per second and reports saving 2 engineers' worth of ongoing infrastructure maintenance time.

June 26, 2025Thursday
June 24, 2025Tuesday
  1. Vapi Blog22

    Mistral vs Llama 3: A Complete Comparison for Voice AI Applications

    Mistral and Llama 3 split on philosophy for voice AI: Mistral's 7B to Mistral Small 3.1 (24B) models favor low-latency efficiency, while Llama 3 scales from 8B to 70B for stronger reasoning and multilingual reach. Mistral Small 3.1 reaches 0.29s time to first token at 150-166 tokens per second, and Mistral Small 24B beats Llama 3.1 8B Instruct on ARC-C, GPQA and MMLU, while Llama 3.1 405B scores 85.2% on MMLU and 96.8% on GSM8K.

June 23, 2025Monday
  1. Vapi Blog22

    8 Alternatives to Azure for Voice AI STT on Vapi

    Vapi now offers eight speech-to-text providers — Gladia, AssemblyAI, Deepgram, Cartesia, Talkscriber, OpenAI, Speechmatics and ElevenLabs — as drop-in alternatives to Azure for voice agents. Switching providers takes a single JSON parameter change, letting developers benchmark latency, cost and compliance. Gladia claims sub-100 ms end-to-end latency, AssemblyAI bundles summaries and PII redaction in one API call, and Cartesia runs fully on-premises.

June 19, 2025Thursday
  1. Vapi Blog22

    Gemini Flash vs Pro: Comparing Google's Latest LLMs

    Google's Gemini 2.5 Flash and 2.5 Pro differ mainly in speed versus reasoning depth, both offering 1M-token context windows and native availability in Vapi for voice agents. Flash returns its first token in 0.21–0.37 seconds at 163 tokens per second and costs $0.15/$0.60 per million input/output tokens, while Pro costs $1.25/$10 and adds a Deep Think mode, with a 2M-token window planned.

June 18, 2025Wednesday
  1. Modal Blog58

    Modal shows how to run FLUX.1-dev three times faster

    Modal engineers describe making Black Forest Labs' FLUX.1-dev run about three times faster as an autoscaling service, cutting a 1024x1024 image from roughly 6.75 seconds to under three. Standard optimizations, namely the Torch compiler with max-autotune, fused QKV attention and channels-last memory layout, gave about 1.5x, while First Block Caching from the ParaAttention repo, based on the TEACache paper, added another 2x at a threshold of 0.12. Compile caches and Modal Memory Snapshots cut cold start latency by 30x.

June 17, 2025Tuesday
  1. Vapi Blog38

    Deepgram Nova-3 vs Nova-2: How the STT Engine Evolved

    Deepgram's Nova-3 cuts streaming word error rate from Nova-2's 8.4% to 6.84% and reaches 5.26% batch WER, a 54.3% reduction in streaming errors. Nova-3 replaces Nova-2's separate domain-specific models with a single adaptable network and adds real-time multilingual conversation handling, up to 100 self-serve custom terms without retraining, and redaction for up to 50 entities. Both models run natively on Vapi with identical endpoints, so switching is a parameter change, though Nova-3 sits at Deepgram's premium pricing tier.

June 9, 2025Monday
  1. Modal Blog40

    Modal 1.0 Released: More Stable Python SDK with Explicit Configuration

    Modal has launched version 1.0 of its Python SDK client, a milestone the company says brings a new level of maturity and stability to its serverless cloud compute platform, with far fewer breaking changes expected going forward. The release removes "magic" behaviors such as automatic mounting of local packages, splits core functionality across separate decorators like the new @modal.concurrent, and consolidates local file handling under the modal.Image class. Modal also plans SDKs for other languages and is exploring how the platform should serve AI-driven agent workflows.

June 4, 2025Wednesday
  1. Vapi Blog22

    Three Examples of Voice AI Improving Logistics

    Vapi outlines three voice AI use cases in logistics: ShipEX uses voice intelligence across 350 trucks to route maintenance alerts and accident reports, the Idaho State Liquor Division's voice picking system lifted pick rates 35%, and maintenance technicians log issues by voice to auto-create CMMS work orders. The article also details API integration with TMS, CRM, and WMS systems and deployment considerations.

  2. Vapi Blog22

    11 Great ElevenLabs Alternatives: Vapi-Native TTS Models

    Vapi now offers 11 built-in text-to-speech models as alternatives to ElevenLabs, including Neuphonic's Neu-hq and Neu-fast, Cartesia's Sonic lineup, Azure TTS, OpenAI TTS, Deepgram, Smallest AI's Lightning, LMNT, PlayHT, Hume's Octave and Rime AI's Mist v2. Developers can switch between them in minutes without re-coding, and Vapi says it handles over a million voice calls per day across these models.