Modal profiled SGLang's scheduler on a multimodal workload and replaced repeated CUDA IPC handle book-keeping with a simple Python dict cache, lifting throughput 16.2% and cutting mean TTFT 13.2% and TPOT 17.2% on Qwen2.5-VL-3B-Instruct on H100s. The fix is merged in SGLang v0.5.10.
Vapi Blog interviews Cartesia's Iz Shalom on choosing between integrated and open modular voice AI stacks. Integrated stacks suit narrow use cases and fast proofs of concept, while open stacks fit multiple use cases, multilingual support, and swapping components. The piece urges teams to measure P99 latency in their own environment and build tight evaluations before production.
AE Studio details how it trained LLMs to prove math theorems with reinforcement learning on Modal, comparing Evolution Strategies against GRPO while keeping the training loop fixed. The setup runs vLLM inference on GPUs, Lean compiler verification in isolated CPU sandboxes, and a lightweight orchestrator, using per-function images, .map() fan-out, and Volumes for base weights.
OpenAI published an API deployment checklist that walks developers through moving workloads onto the Responses API and the GPT-6 model family, including gpt-6-astra, gpt-6.1-sol, gpt-6 Sol and gpt-6 Luna. The guide covers reasoning.effort levels from low to max, text.verbosity, tool_search with deferred tool loading, Programmatic Tool Calling, Multi-agent, compaction, prompt caching and WebSocket mode, which OpenAI says has shown up to roughly 40% faster end-to-end execution for workflows with 20 or more tool calls. It also details reliability and safety steps such as handling 429 and 503 errors, background=True for long jobs, safety_identifier and misalignment monitoring.
Why it matters: OpenAI's official deployment checklist consolidates model selection, reasoning effort, tool search, compaction and WebSocket mode into one migration guide for GPT-6 workloads.
Modal published a walkthrough on building a custom coding agent harness on top of the OpenAI Agents SDK, which OpenAI launched today, using Modal Sandboxes to give agents isolated Linux environments and GPUs. The guide starts from a minimal exec-based agent, then adds sessions for memory, an orchestrator plus subagent pool for parallel async work, GPU quotas, filesystem snapshots to deduplicate setup, and a skills subsystem, ending with a harness that runs OpenAI's Parameter Golf challenge across parallel GPU subagents. Modal cites Ramp's fleet of background coding agents on Modal, which it says now produces more than half of the company's PRs.
Vapi's Forward-Deployed Engineers say the demo-to-production gap is where most enterprise voice AI initiatives stall, requiring extensive testing, guardrails and intent-based design rather than exact phrase matching. The team warns of "personality drift," where agents adopt callers' speech styles, and notes LLMs are non-deterministic, so deployments need defined error tolerance and continuous iteration. They advise starting with cost optimization but targeting revenue generation, and benchmark a good interaction at under a minute.
Doppel migrated a large portion of its ML workflows to Modal, removing the infrastructure friction that slowed training experimentation and inference deployment. Parallel experiments via map() replaced sequential runs, while image layer caching and persistent volumes cut build times by up to 10x, with complex setups reaching sub-minute warm builds.
OpenAI updated its Sora 2 prompting guide to cover new API capabilities: reusable character references for objects and animals, 1920×1080 and 1080×1920 exports, a maximum clip length raised from 12s to 20s, video extension that uses the full original clip as context, and a Batch API for asynchronous video jobs. The guide also walks through prompt structure for shot framing, camera motion, lighting and dialogue, and notes that resolution, duration and character IDs must be set as API parameters rather than described in prose. Extensions of up to 20 seconds are supported, and a video can be extended up to 6 times for a total of 120s.
Why it matters: OpenAI's updated Sora 2 prompting guide details new API capabilities such as character references, 20s clips and video extension, plus reusable prompt templates for controlling shots.
OpenAI released a Codex prompting guide for developers calling the Codex-tuned gpt-5.3-codex model directly through the API, recommending medium reasoning effort for interactive coding and high or xhigh for the hardest tasks. The guide covers migrating an existing harness to the open-source codex-cli, including the standard Codex-Max prompt, the apply_patch, shell_command and update_plan tools, and first-class compaction via /compact in the Responses API for multi-hour sessions. It also describes the new phase field, supported only on gpt-5.3-codex, which marks assistant items as commentary or final_answer and must be preserved in history to avoid performance degradation.
Why it matters: OpenAI's official guide details how to tune prompts, tools and compaction for gpt-5.3-codex, giving teams a concrete migration path for their own coding agents.
OpenClaw agents can now make and receive phone calls through Vapi's AgentSkills-compatible skills, installed in about five minutes via the npx skills CLI or by asking the agent directly. The Vapi pipeline targets sub-500ms voice responses and supports mid-call tool calling, with eight skills covering assistants, calls, phone numbers, tools, squads, webhooks and workflows.
Ramp built Ramp Inspect, an internal background coding agent running on Modal Sandboxes, and it now starts roughly half of all merged pull requests across Ramp's frontend and backend repos. Each session boots a full-stack environment with Postgres, Redis, Temporal and RabbitMQ in seconds, because a Modal cron job clones repos and saves filesystem snapshots every 30 minutes. Modal Functions, Dicts and Queues handle snapshot builds, session locks and prompt routing, letting Inspect support multiplayer sessions and hundreds of concurrent sessions without one affecting another.
Modal has scaled its multi-cloud GPU worker pool past 20,000 concurrent GPUs and launched over four million cloud instances in the last couple years, and is now sharing the reliability system behind it. The post covers instance type testing and selection, machine image preparation and boot checks, passive and active GPU healthchecking, and observability and support. It reports hyperscaler differences such as Cloud A's 99.6% boot success and Cloud C's H100s running above 90ºC in 2025.
Reducto migrated inference workloads for its 30+ document-processing models to Modal, cutting P90 latency by 3x. GPU memory snapshotting reduced cold boots by 83%, from ~70s to ~12s, while per-customer autoscaling pools isolate traffic spikes. In a load test, Reducto scaled its ingestion pipeline to over 1,000 GPUs in under an hour to demonstrate 100k pages per minute.
Modal's blog explains how host overhead—gaps where the GPU sits idle waiting for the CPU to prepare work—shows up as low GPU kernel utilization in AI inference. At 50% host overhead, GPU inference costs are 2x what they could be. The post details fixes used in production open source inference engines, including constructing tensors on-device instead of transferring them and reducing unnecessary CUDA kernel launches.
Decagon and Modal partnered to build Decagon Voice 2.0, cutting latency by 65% while improving intent recognition and response quality. Decagon fine-tuned compact open-source models with SFT and RL to surpass larger frontier models, while Modal trained a custom EAGLE3 draft model with 38% higher accept lengths and re-engineered SGLang with an async speculative decoding scheduler, boosting throughput up to 12%.
Modal's Ben Shababo walks through building a conversational voice bot on Modal with the open source Pipecat framework and open weights models, reaching a median voice-to-voice latency of about one second when the client and containers are geographically close. The pipeline uses NVIDIA parakeet-tdt-0.6b-v3 for STT, Qwen3-4B-Instruct-2507 served with vLLM, KokoroTTS for speech output, and ChromaDB with all-minilm-l6-v2 for RAG over Modal's docs. Modal Tunnels bypass the input plane for low-latency WebSocket and HTTP links to the STT, TTS and vLLM services, while a FunctionCall tied to each conversation session recovers autoscaling, and pinning services to nearby regions cuts network latency.
Modal engineers read the released Flash Attention 4 source code and reverse-engineered how the Blackwell-optimized attention kernel works, since no official technical report exists yet. The write-up traces a tile through the kernel's warp-specialized asynchronous pipeline and details two math tricks: a cubic-polynomial software exponential that can replace SFU-based exponentiation, and a smarter online softmax rescaling rule that Tri Dao said cuts corrections by a factor of 10. FA4 is reported to be roughly 20% faster than Nvidia's closed-source cudnn attention kernels, and a backwards version was not yet available at the time of writing.
Why it matters: Modal engineers read the released Flash Attention 4 source and explain its warp-specialized pipeline and two math tricks, offering a transferable view of where CUDA kernel design is heading.
Modal detailed the engineering behind Modal Notebooks, its cloud Jupyter notebook that boots GPUs and custom images in seconds with real-time collaboration. Kernels run inside Modal Sandboxes, with a modal-kernelshim daemon translating Jupyter ZeroMQ messages into HTTP calls tunneled through Modal's control plane. A lazy-loading Rust FUSE container filesystem fetches file contents on demand through a tiered cache, avoiding the image-unpacking latency that can approach a minute for an 8 GB Python/ML image.
Zencastr moved its podcast transcription workloads to Modal's GPU platform, scaling to 1,500 concurrent GPUs within minutes to process hundreds of years of audio in just a few days. The shift from self-managed Kubernetes GPU nodes to Modal's scale-to-zero infrastructure cut idle GPU costs and removed driver and dependency management overhead. All core ML workloads now run on Modal, including transcription, speaker counting, laughter detection, and audio quality scoring.
An AI code sandbox is an isolated environment that executes LLM-generated code at runtime, containing the blast radius of errors or injected vulnerabilities. Modern implementations run on VMs or containers, often using secure runtimes like gVisor to intercept syscalls and shield the host. Most companies choose managed sandbox services over building their own with Kubernetes, since cold starts and scaling to millions of concurrent sessions are costly to handle.
Modal implemented a batch transcription service on its platform using NVIDIA's parakeet-tdt-0.6b-v2 and canary-1b-flash, reporting over 100x faster or 100x cheaper end-to-end performance than a popular proprietary ASR API while matching its error rate on roughly a week of ESB benchmark audio. The team attributes the gains to shuffling files across request batches, sorting audio by duration within GPU batches, parallel downloads, and empirically chosen GPU and worker configurations. The post walks through the architecture and optimization steps so others can build similar services.
Modal used evals and inference-time compute scaling to improve its QArt code system, boosting QR scan rate to its 95% service-level objective while improving aesthetic quality and returning codes in under 20s at p95. The work follows the standard ML playbook of collecting human judgments, aligning to them, then scaling, addressing the "it worked on my prompt" gap that leaves many generated QR codes unscannable.
Quora uses Modal Sandboxes to securely execute LLM-generated code in Poe, its AI chatbot platform, shipping the feature months earlier than an in-house build would have allowed. The team stress-tested Sandbox creation throughput to 1000 sandboxes per second and reports saving 2 engineers' worth of ongoing infrastructure maintenance time.
Mistral and Llama 3 split on philosophy for voice AI: Mistral's 7B to Mistral Small 3.1 (24B) models favor low-latency efficiency, while Llama 3 scales from 8B to 70B for stronger reasoning and multilingual reach. Mistral Small 3.1 reaches 0.29s time to first token at 150-166 tokens per second, and Mistral Small 24B beats Llama 3.1 8B Instruct on ARC-C, GPQA and MMLU, while Llama 3.1 405B scores 85.2% on MMLU and 96.8% on GSM8K.
Vapi now offers eight speech-to-text providers — Gladia, AssemblyAI, Deepgram, Cartesia, Talkscriber, OpenAI, Speechmatics and ElevenLabs — as drop-in alternatives to Azure for voice agents. Switching providers takes a single JSON parameter change, letting developers benchmark latency, cost and compliance. Gladia claims sub-100 ms end-to-end latency, AssemblyAI bundles summaries and PII redaction in one API call, and Cartesia runs fully on-premises.
Google's Gemini 2.5 Flash and 2.5 Pro differ mainly in speed versus reasoning depth, both offering 1M-token context windows and native availability in Vapi for voice agents. Flash returns its first token in 0.21–0.37 seconds at 163 tokens per second and costs $0.15/$0.60 per million input/output tokens, while Pro costs $1.25/$10 and adds a Deep Think mode, with a 2M-token window planned.
Modal engineers describe making Black Forest Labs' FLUX.1-dev run about three times faster as an autoscaling service, cutting a 1024x1024 image from roughly 6.75 seconds to under three. Standard optimizations, namely the Torch compiler with max-autotune, fused QKV attention and channels-last memory layout, gave about 1.5x, while First Block Caching from the ParaAttention repo, based on the TEACache paper, added another 2x at a threshold of 0.12. Compile caches and Modal Memory Snapshots cut cold start latency by 30x.
A guide outlines five ways real estate agents can use AI, including voice agents built on Vapi that answer calls, qualify leads, and book showings around the clock. It also covers automating content with ChatGPT and Canva, plus AI pricing tools such as HouseCanary and CoreLogic that produce valuations with confidence scores.
Deepgram's Nova-3 cuts streaming word error rate from Nova-2's 8.4% to 6.84% and reaches 5.26% batch WER, a 54.3% reduction in streaming errors. Nova-3 replaces Nova-2's separate domain-specific models with a single adaptable network and adds real-time multilingual conversation handling, up to 100 self-serve custom terms without retraining, and redaction for up to 50 entities. Both models run natively on Vapi with identical endpoints, so switching is a parameter change, though Nova-3 sits at Deepgram's premium pricing tier.
ElevenLabs Flash v2.5 reaches 75ms first-audio latency versus OpenAI's 200ms, while OpenAI charges $15 per million characters against ElevenLabs' $5–$1,320 monthly plans. ElevenLabs offers 3,000+ voices with cloning and 82% word accuracy, compared with OpenAI's 11 fixed voices at 77%. Both TTS providers are built into Vapi for direct testing.
Vapi outlines three voice AI use cases in logistics: ShipEX uses voice intelligence across 350 trucks to route maintenance alerts and accident reports, the Idaho State Liquor Division's voice picking system lifted pick rates 35%, and maintenance technicians log issues by voice to auto-create CMMS work orders. The article also details API integration with TMS, CRM, and WMS systems and deployment considerations.
Vapi outlines how to pick among four Gemini models for voice agents: Gemini 1.0 Pro for predictable, compliance-friendly flows at $0.50 input and $2.00 output per million tokens, Gemini 1.5 Flash for high-volume work with a 1M token context window, Gemini 1.5 Pro for deep reasoning with a 2M token window and 99.7% long-context retrieval accuracy, and Gemini 2.0 Flash for native tool calling.
DeepSeek V3 processes 47% more tokens per second than R1 at $0.28 per million output tokens, while R1 costs $2.19 per million and runs internal reasoning loops for harder tasks. R1 scores 97.3% on MATH-500 and 79.8% on AIME 2024 versus V3's 90.2% and 39.2%. Most teams route 80-90% of simple queries to V3 and send complex reasoning work to R1.
SRTP secures RTP voice and video streams by adding encrypted payloads, authentication tags, and a master key identifier to each media packet. It relies primarily on AES counter mode and HMAC-SHA1 authentication tags, with sequence numbers blocking replay attacks, and pairs with DTLS-SRTP or SDES for key exchange. SRTP is positioned as a lower-overhead, application-layer option for real-time communications, with GDPR, HIPAA, and CPRA cited as compliance drivers.
A guide on using .env files and environment configuration in voice AI projects, covering dotenv setup for Node.js and Python, adding .env to .gitignore, and Vapi-specific keys like VAPI_API_KEY and VAPI_PROJECT_ID. It cites GitGuardian's finding of 12.8 million secrets leaked on GitHub in 2023, a 28% increase, and recommends separate env files per environment plus tools such as git-crypt, SOPS, AWS Secrets Manager and HashiCorp Vault.
Amazon Lex suits teams already on AWS needing tight integration with Amazon services, while Dialogflow offers broader language support and an easier learning curve for newcomers. Lex uses Alexa's technology with pay-as-you-go pricing including a free tier of 10,000 text and 5,000 speech requests monthly for the first year, and Dialogflow supports over 30 languages with pre-built agents and sentiment analysis.
Vapi's guide explains how to configure environment variables for its Voice AI integration, covering temporary vs. permanent and user-level vs. system-level variables. It details setup on Windows, macOS and Linux with Vapi-specific examples like VAPI_API_KEY and VAPI_MULTILINGUAL_SUPPORT, plus naming and security best practices such as never hardcoding API keys.
A guide ranks 10 open-source speech-to-text models for clinical use, evaluated on real-time accuracy, clinical terminology handling, operational reliability and data security. The list covers Vosk, DeepSpeech, SpeechBrain, OpenAI Whisper and ESPnet, matched to settings such as low-resource rural clinics, multilingual hospitals and terminology-heavy research centers. All can be integrated with Vapi's model-agnostic voice AI platform.
Modal built a "resource solver" that uses linear programming and the simplex algorithm to decide which cloud instances to spin up or down, meeting demand for GPUs at predictable prices. The system also captures temporary arbitrage, such as hundreds of H200 GPUs at 20% below the going rate for H100s, saving millions of dollars a year.
sync. is using Modal to lipsync over 100 hours of video a day, splitting long videos into scenes that run face detection and translation across n T4 GPU containers before passing each scene to its proprietary LipSync GenAI model on A100s. The team deploys up to 95 times a day and shipped 10 major model variants to production in one year, after moving off a Google Colab prototype and AWS Lambda.