Skip to content
#Tutorials / practice
February 25Wednesday
  1. OpenAI Developers Blog62

    OpenAI publishes Codex prompting guide for gpt-5.3-codex

    OpenAI released a Codex prompting guide for developers calling the Codex-tuned gpt-5.3-codex model directly through the API, recommending medium reasoning effort for interactive coding and high or xhigh for the hardest tasks. The guide covers migrating an existing harness to the open-source codex-cli, including the standard Codex-Max prompt, the apply_patch, shell_command and update_plan tools, and first-class compaction via /compact in the Responses API for multi-hour sessions. It also describes the new phase field, supported only on gpt-5.3-codex, which marks assistant items as commentary or final_answer and must be preserved in history to avoid performance degradation.

    Why it matters: OpenAI's official guide details how to tune prompts, tools and compaction for gpt-5.3-codex, giving teams a concrete migration path for their own coding agents.

February 24Tuesday
  1. Vapi Blog38

    Give Your OpenClaw Agent a Voice: Adding Phone Calls with Vapi Skills

    OpenClaw agents can now make and receive phone calls through Vapi's AgentSkills-compatible skills, installed in about five minutes via the npx skills CLI or by asking the agent directly. The Vapi pipeline targets sub-500ms voice responses and supports mid-call tool calling, with eight skills covering assistants, calls, phone numbers, tools, squads, webhooks and workflows.

February 19Thursday
  1. Modal Blog58

    How Ramp built Ramp Inspect, a full-context background coding agent on Modal

    Ramp built Ramp Inspect, an internal background coding agent running on Modal Sandboxes, and it now starts roughly half of all merged pull requests across Ramp's frontend and backend repos. Each session boots a full-stack environment with Postgres, Redis, Temporal and RabbitMQ in seconds, because a Modal cron job clones repos and saves filesystem snapshots every 30 minutes. Modal Functions, Dicts and Queues handle snapshot builds, session locks and prompt routing, letting Inspect support multiplayer sessions and hundreds of concurrent sessions without one affecting another.

December 28, 2025Sunday
  1. Modal Blog38

    Modal on Keeping 20,000+ GPUs Healthy Across Clouds

    Modal has scaled its multi-cloud GPU worker pool past 20,000 concurrent GPUs and launched over four million cloud instances in the last couple years, and is now sharing the reliability system behind it. The post covers instance type testing and selection, machine image preparation and boot checks, passive and active GPU healthchecking, and observability and support. It reports hyperscaler differences such as Cloud A's 99.6% boot success and Cloud C's H100s running above 90ºC in 2025.

November 19, 2025Wednesday
  1. Modal Blog30

    How Reducto Cut Enterprise Document Processing P90 Latency 3x on Modal

    Reducto migrated inference workloads for its 30+ document-processing models to Modal, cutting P90 latency by 3x. GPU memory snapshotting reduced cold boots by 83%, from ~70s to ~12s, while per-customer autoscaling pools isolate traffic spikes. In a load test, Reducto scaled its ingestion pipeline to over 1,000 GPUs in under an hour to demonstrate 100k pages per minute.

November 18, 2025Tuesday
  1. Modal Blog36

    Modal Blog: Host Overhead Is Killing Your Inference Efficiency

    Modal's blog explains how host overhead—gaps where the GPU sits idle waiting for the CPU to prepare work—shows up as low GPU kernel utilization in AI inference. At 50% host overhead, GPU inference costs are 2x what they could be. The post details fixes used in production open source inference engines, including constructing tensors on-device instead of transferring them and reducing unnecessary CUDA kernel launches.

November 13, 2025Thursday
  1. Modal Blog40

    How Decagon Shipped Real-Time Voice AI on Modal

    Decagon and Modal partnered to build Decagon Voice 2.0, cutting latency by 65% while improving intent recognition and response quality. Decagon fine-tuned compact open-source models with SFT and RL to surpass larger frontier models, while Modal trained a custom EAGLE3 draft model with 38% higher accept lengths and re-engineered SGLang with an async speculative decoding scheduler, boosting throughput up to 12%.

November 4, 2025Tuesday
  1. Modal Blog58

    Modal, Pipecat and open models achieve one-second voice-to-voice latency

    Modal's Ben Shababo walks through building a conversational voice bot on Modal with the open source Pipecat framework and open weights models, reaching a median voice-to-voice latency of about one second when the client and containers are geographically close. The pipeline uses NVIDIA parakeet-tdt-0.6b-v3 for STT, Qwen3-4B-Instruct-2507 served with vLLM, KokoroTTS for speech output, and ChromaDB with all-minilm-l6-v2 for RAG over Modal's docs. Modal Tunnels bypass the input plane for low-latency WebSocket and HTTP links to the STT, TTS and vLLM services, while a FunctionCall tied to each conversation session recovers autoscaling, and pinning services to nearby regions cuts network latency.

September 26, 2025Friday
  1. Modal Blog60

    Modal engineers reverse-engineer how Flash Attention 4 works

    Modal engineers read the released Flash Attention 4 source code and reverse-engineered how the Blackwell-optimized attention kernel works, since no official technical report exists yet. The write-up traces a tile through the kernel's warp-specialized asynchronous pipeline and details two math tricks: a cubic-polynomial software exponential that can replace SFU-based exponentiation, and a smarter online softmax rescaling rule that Tri Dao said cuts corrections by a factor of 10. FA4 is reported to be roughly 20% faster than Nvidia's closed-source cudnn attention kernels, and a backwards version was not yet available at the time of writing.

    Why it matters: Modal engineers read the released Flash Attention 4 source and explain its warp-specialized pipeline and two math tricks, offering a transferable view of where CUDA kernel design is heading.

September 16, 2025Tuesday
  1. Modal Blog47

    Inside Modal Notebooks: How Modal Built a Cloud GPU Notebook That Boots in Seconds

    Modal detailed the engineering behind Modal Notebooks, its cloud Jupyter notebook that boots GPUs and custom images in seconds with real-time collaboration. Kernels run inside Modal Sandboxes, with a modal-kernelshim daemon translating Jupyter ZeroMQ messages into HTTP calls tunneled through Modal's control plane. A lazy-loading Rust FUSE container filesystem fetches file contents on demand through a tiered cache, avoiding the image-unpacking latency that can approach a minute for an 8 GB Python/ML image.

August 28, 2025Thursday
  1. Modal Blog22

    How Zencastr transcribed hundreds of years of audio in days with Modal

    Zencastr moved its podcast transcription workloads to Modal's GPU platform, scaling to 1,500 concurrent GPUs within minutes to process hundreds of years of audio in just a few days. The shift from self-managed Kubernetes GPU nodes to Modal's scale-to-zero infrastructure cut idle GPU costs and removed driver and dependency management overhead. All core ML workloads now run on Modal, including transcription, speaker counting, laughter detection, and audio quality scoring.

July 24, 2025Thursday
  1. Modal Blog36

    What is an AI code sandbox?

    An AI code sandbox is an isolated environment that executes LLM-generated code at runtime, containing the blast radius of errors or injected vulnerabilities. Modern implementations run on VMs or containers, often using secure runtimes like gVisor to intercept syscalls and shield the host. Most companies choose managed sandbox services over building their own with Kubernetes, since cold starts and scaling to millions of concurrent sessions are costly to handle.

July 23, 2025Wednesday
  1. Modal Blog52

    Modal benchmarks open ASR models for batch transcription at 100x speed and cost gains

    Modal implemented a batch transcription service on its platform using NVIDIA's parakeet-tdt-0.6b-v2 and canary-1b-flash, reporting over 100x faster or 100x cheaper end-to-end performance than a popular proprietary ASR API while matching its error rate on roughly a week of ESB benchmark audio. The team attributes the gains to shuffling files across request batches, sorting audio by duration within GPU batches, parallel downloads, and empirically chosen GPU and worker configurations. The post walks through the architecture and optimization steps so others can build similar services.

July 2, 2025Wednesday
  1. Modal Blog40

    How Modal Used Evals and Inference-Time Compute Scaling to Generate Scannable QArt Codes

    Modal used evals and inference-time compute scaling to improve its QArt code system, boosting QR scan rate to its 95% service-level objective while improving aesthetic quality and returning codes in under 20s at p95. The work follows the standard ML playbook of collecting human judgments, aligning to them, then scaling, addressing the "it worked on my prompt" gap that leaves many generated QR codes unscannable.

June 30, 2025Monday
  1. Modal Blog28

    How Quora uses Modal to run thousands of Python sandboxes simultaneously

    Quora uses Modal Sandboxes to securely execute LLM-generated code in Poe, its AI chatbot platform, shipping the feature months earlier than an in-house build would have allowed. The team stress-tested Sandbox creation throughput to 1000 sandboxes per second and reports saving 2 engineers' worth of ongoing infrastructure maintenance time.

June 24, 2025Tuesday
  1. Vapi Blog22

    Mistral vs Llama 3: A Complete Comparison for Voice AI Applications

    Mistral and Llama 3 split on philosophy for voice AI: Mistral's 7B to Mistral Small 3.1 (24B) models favor low-latency efficiency, while Llama 3 scales from 8B to 70B for stronger reasoning and multilingual reach. Mistral Small 3.1 reaches 0.29s time to first token at 150-166 tokens per second, and Mistral Small 24B beats Llama 3.1 8B Instruct on ARC-C, GPQA and MMLU, while Llama 3.1 405B scores 85.2% on MMLU and 96.8% on GSM8K.

June 23, 2025Monday
  1. Vapi Blog22

    8 Alternatives to Azure for Voice AI STT on Vapi

    Vapi now offers eight speech-to-text providers — Gladia, AssemblyAI, Deepgram, Cartesia, Talkscriber, OpenAI, Speechmatics and ElevenLabs — as drop-in alternatives to Azure for voice agents. Switching providers takes a single JSON parameter change, letting developers benchmark latency, cost and compliance. Gladia claims sub-100 ms end-to-end latency, AssemblyAI bundles summaries and PII redaction in one API call, and Cartesia runs fully on-premises.

June 19, 2025Thursday
  1. Vapi Blog22

    Gemini Flash vs Pro: Comparing Google's Latest LLMs

    Google's Gemini 2.5 Flash and 2.5 Pro differ mainly in speed versus reasoning depth, both offering 1M-token context windows and native availability in Vapi for voice agents. Flash returns its first token in 0.21–0.37 seconds at 163 tokens per second and costs $0.15/$0.60 per million input/output tokens, while Pro costs $1.25/$10 and adds a Deep Think mode, with a 2M-token window planned.

June 18, 2025Wednesday
  1. Modal Blog58

    Modal shows how to run FLUX.1-dev three times faster

    Modal engineers describe making Black Forest Labs' FLUX.1-dev run about three times faster as an autoscaling service, cutting a 1024x1024 image from roughly 6.75 seconds to under three. Standard optimizations, namely the Torch compiler with max-autotune, fused QKV attention and channels-last memory layout, gave about 1.5x, while First Block Caching from the ParaAttention repo, based on the TEACache paper, added another 2x at a threshold of 0.12. Compile caches and Modal Memory Snapshots cut cold start latency by 30x.

  2. Vapi Blog18

    Five Ways Real Estate Agents Can Use AI

    A guide outlines five ways real estate agents can use AI, including voice agents built on Vapi that answer calls, qualify leads, and book showings around the clock. It also covers automating content with ChatGPT and Canva, plus AI pricing tools such as HouseCanary and CoreLogic that produce valuations with confidence scores.

June 17, 2025Tuesday
  1. Vapi Blog38

    Deepgram Nova-3 vs Nova-2: How the STT Engine Evolved

    Deepgram's Nova-3 cuts streaming word error rate from Nova-2's 8.4% to 6.84% and reaches 5.26% batch WER, a 54.3% reduction in streaming errors. Nova-3 replaces Nova-2's separate domain-specific models with a single adaptable network and adds real-time multilingual conversation handling, up to 100 self-serve custom terms without retraining, and redaction for up to 50 entities. Both models run natively on Vapi with identical endpoints, so switching is a parameter change, though Nova-3 sits at Deepgram's premium pricing tier.

June 4, 2025Wednesday
  2. Vapi Blog22

    Three Examples of Voice AI Improving Logistics

    Vapi outlines three voice AI use cases in logistics: ShipEX uses voice intelligence across 350 trucks to route maintenance alerts and accident reports, the Idaho State Liquor Division's voice picking system lifted pick rates 35%, and maintenance technicians log issues by voice to auto-create CMMS work orders. The article also details API integration with TMS, CRM, and WMS systems and deployment considerations.

May 29, 2025Thursday
  1. Vapi Blog25

    Choosing Between Gemini Models for Voice AI

    Vapi outlines how to pick among four Gemini models for voice agents: Gemini 1.0 Pro for predictable, compliance-friendly flows at $0.50 input and $2.00 output per million tokens, Gemini 1.5 Flash for high-volume work with a 1M token context window, Gemini 1.5 Pro for deep reasoning with a 2M token window and 99.7% long-context retrieval accuracy, and Gemini 2.0 Flash for native tool calling.

May 28, 2025Wednesday
May 26, 2025Monday
  1. Vapi Blog10

    Introduction to SRTP: Secure Real-Time Transport Protocol

    SRTP secures RTP voice and video streams by adding encrypted payloads, authentication tags, and a master key identifier to each media packet. It relies primarily on AES counter mode and HMAC-SHA1 authentication tags, with sequence numbers blocking replay attacks, and pairs with DTLS-SRTP or SDES for key exchange. SRTP is positioned as a lower-overhead, application-layer option for real-time communications, with GDPR, HIPAA, and CPRA cited as compliance drivers.

  2. Vapi Blog18

    Env Files and Environment Variables for Voice AI Projects

    A guide on using .env files and environment configuration in voice AI projects, covering dotenv setup for Node.js and Python, adding .env to .gitignore, and Vapi-specific keys like VAPI_API_KEY and VAPI_PROJECT_ID. It cites GitGuardian's finding of 12.8 million secrets leaked on GitHub in 2023, a 28% increase, and recommends separate env files per environment plus tools such as git-crypt, SOPS, AWS Secrets Manager and HashiCorp Vault.

May 23, 2025Friday
  1. Vapi Blog18

    Amazon Lex vs Dialogflow: Complete Platform Comparison Guide

    Amazon Lex suits teams already on AWS needing tight integration with Amazon services, while Dialogflow offers broader language support and an easier learning curve for newcomers. Lex uses Alexa's technology with pay-as-you-go pricing including a free tier of 10,000 text and 5,000 speech requests monthly for the first year, and Dialogflow supports over 30 languages with pre-built agents and sentiment analysis.

  2. Vapi Blog8

    Mastering Environment Variables for Vapi Voice AI Integration

    Vapi's guide explains how to configure environment variables for its Voice AI integration, covering temporary vs. permanent and user-level vs. system-level variables. It details setup on Windows, macOS and Linux with Vapi-specific examples like VAPI_API_KEY and VAPI_MULTILINGUAL_SUPPORT, plus naming and security best practices such as never hardcoding API keys.

May 22, 2025Thursday
  1. Vapi Blog20

    The 10 Best Open-Source Medical Speech-to-Text Tools

    A guide ranks 10 open-source speech-to-text models for clinical use, evaluated on real-time accuracy, clinical terminology handling, operational reliability and data security. The list covers Vosk, DeepSpeech, SpeechBrain, OpenAI Whisper and ESPnet, matched to settings such as low-resource rural clinics, multilingual hospitals and terminology-heavy research centers. All can be integrated with Vapi's model-agnostic voice AI platform.

May 7, 2025Wednesday
April 18, 2025Friday
  1. Modal Blog22

    How sync. uses Modal to lipsync 100 hours of video a day

    sync. is using Modal to lipsync over 100 hours of video a day, splitting long videos into scenes that run face detection and translation across n T4 GPU containers before passing each scene to its proprietary LipSync GenAI model on A100s. The team deploys up to 95 times a day and shipped 10 major model variants to production in one year, after moving off a Google Colab prototype and AWS Lambda.

April 1, 2025Tuesday
  1. Vapi Blog20

    Scaling Client Intake with Vapi Voice AI Agents: A Consultant's Case Study

    Consultant Christian Ulstrup of GSD at Work used Vapi voice AI agents to replace scheduled discovery calls with an asynchronous voice intake system for his AI Action Sessions workshops. Participants call a dedicated Vapi phone line at their convenience, and transcripts plus AI-generated summaries flow into Slack via Zapier and are logged in HubSpot. For one recent client, 20 participants across the globe contributed over 3 hours of discovery, and the session earned a 5 out of 5 rating.

February 24, 2025Monday
  1. Modal Blog40

    A high-level guide to GPU utilization: allocation, kernel, and Model FLOP/s utilization

    Modal published a guide distinguishing three things commonly called "GPU utilization": GPU Allocation Utilization (GPU-seconds running application code over GPU-seconds paid for), GPU Kernel Utilization (fraction of time running code on GPUs), and Model FLOP/s Utilization (share of theoretical arithmetic bandwidth used). The guide focuses on neural network inference workloads, which Modal calls a revenue center rather than a cost center.

December 10, 2024Tuesday
  1. Modal Blog30

    What is LLM fine-tuning? A Guide to Techniques and Tools

    Fine-tuning adjusts pretrained LLM weights to fit a specific task or domain, delivering higher quality than prompt engineering at lower cost and latency; a fine-tuned Llama 7B can be roughly 50 times more cost-effective per token than GPT-3.5 with comparable performance. The guide covers use cases such as structured output, style adherence and domain-specific instruction, plus options including OpenAI, Predibase, Modal, Google Colab and AWS SageMaker, and frameworks HuggingFace transformers, TRL and Axolotl.

November 20, 2024Wednesday
September 24, 2024Tuesday
September 18, 2024Wednesday
August 5, 2024Monday
  1. Modal Blog62

    Modal replicates Large Language Monkeys: LLaMA 3.1 8B matches GPT-4o on HumanEval via search

    Modal's Charles Frye replicated the core claim of the Large Language Monkeys paper on a new dataset, showing that repeated sampling lifts LLaMA 3.1 8B from 66.4% pass@1 to 90.5% pass@100 and 95.1% pass@1000 on HumanEval, versus GPT-4o's reported 90.2% pass@1. The runs used vLLM on Modal serverless GPUs and Modal Sandboxes for parallel test execution, cost well under $50, and showed smooth, predictable scaling across three orders of magnitude.

    Why it matters: A hands-on replication shows how repeated sampling with a small open model can approach GPT-4o on HumanEval, offering a concrete, low-cost recipe for trading compute for accuracy.

June 20, 2024Thursday
  1. Modal Blog25

    Run GPU jobs from Airflow with Modal

    Modal can now be triggered directly from an Airflow DAG, letting teams run GPU-backed AI/ML tasks without setting up Celery or Kubernetes executors. The guide covers installing Modal in an Airflow environment, setting MODAL_TOKEN_ID and MODAL_TOKEN_SECRET, and two approaches: calling deployed Modal Functions via lookup, or building a custom operator that runs Python in Modal Sandboxes. A Modal Airflow provider package is in progress.