Skip to content
Only First-party#Deployment / engineering
July 23, 2025Wednesday
  1. Modal Blog52

    Modal benchmarks open ASR models for batch transcription at 100x speed and cost gains

    Modal implemented a batch transcription service on its platform using NVIDIA's parakeet-tdt-0.6b-v2 and canary-1b-flash, reporting over 100x faster or 100x cheaper end-to-end performance than a popular proprietary ASR API while matching its error rate on roughly a week of ESB benchmark audio. The team attributes the gains to shuffling files across request batches, sorting audio by duration within GPU batches, parallel downloads, and empirically chosen GPU and worker configurations. The post walks through the architecture and optimization steps so others can build similar services.

July 16, 2025Wednesday
  1. Modal Blog30

    Dollars per token considered harmful: why self-hosted LLM inference should be priced per request

    A Modal Blog post argues that "dollars per token" pricing, inherited from API providers like OpenAI, Anthropic and Alphabet, is actively harmful for teams running self-hosted LLM inference. It contends that application users care about requests rather than tokens, and that per-request framing is needed to reason about latency, replica counts and compute costs.

July 11, 2025Friday
  1. Modal Blog40

    Modal Product Updates: Multi-node Training Clusters, B200 and H200 GPUs, and Client 1.0

    Modal has launched multi-node training clusters in beta, letting users tap dozens of GPUs across multiple hosts by adding the @clustered decorator, with 3.2 Tbps Infiniband RDMA interconnect for near-linear scaling. The platform now offers NVIDIA B200 and H200 GPUs serverlessly at $6.25/hr and $4.54/hr respectively, claiming up to 2-4× faster LLM inference than H100s. Modal also released Client 1.0, emphasizing API stability, alongside updates including a read-only Volume method and timezone support for Cron schedules.

  2. Modal Blog31

    Jamsocket is joining Modal

    Jamsocket, the backend platform behind the open-source orchestrator Plane, Y-Sweet and ForeverVM, is joining Modal, with co-founders Paul Butler and Taylor Baldwin joining the Modal team. Jamsocket products will keep operating normally while core functionality is migrated to Modal, which the founders say lets AI-centric customers deploy GPU inference, code sandboxes and real-time backends from one platform.

July 7, 2025Monday
  1. Modal Blog28

    How Modal Powered 250,000 Lovable App Creations in a Weekend

    Lovable used Modal Sandboxes to run LLM-generated code during a June 2025 promotional weekend with Anthropic, OpenAI, and Google, enabling users to build an estimated 250,000 applications in 48 hours. Modal ran over 1 million sandboxes with up to 20,000 concurrent at peak as Lovable's concurrent sessions surged 2.5-3x, and its platform team was not paged once. Lovable cut its sandbox orchestration code from 15,000 lines to 700 lines after switching to Modal.

July 2, 2025Wednesday
  1. Modal Blog40

    How Modal Used Evals and Inference-Time Compute Scaling to Generate Scannable QArt Codes

    Modal used evals and inference-time compute scaling to improve its QArt code system, boosting QR scan rate to its 95% service-level objective while improving aesthetic quality and returning codes in under 20s at p95. The work follows the standard ML playbook of collecting human judgments, aligning to them, then scaling, addressing the "it worked on my prompt" gap that leaves many generated QR codes unscannable.

June 30, 2025Monday
  1. Modal Blog28

    How Quora uses Modal to run thousands of Python sandboxes simultaneously

    Quora uses Modal Sandboxes to securely execute LLM-generated code in Poe, its AI chatbot platform, shipping the feature months earlier than an in-house build would have allowed. The team stress-tested Sandbox creation throughput to 1000 sandboxes per second and reports saving 2 engineers' worth of ongoing infrastructure maintenance time.

June 26, 2025Thursday
June 24, 2025Tuesday
  1. Vapi Blog22

    Mistral vs Llama 3: A Complete Comparison for Voice AI Applications

    Mistral and Llama 3 split on philosophy for voice AI: Mistral's 7B to Mistral Small 3.1 (24B) models favor low-latency efficiency, while Llama 3 scales from 8B to 70B for stronger reasoning and multilingual reach. Mistral Small 3.1 reaches 0.29s time to first token at 150-166 tokens per second, and Mistral Small 24B beats Llama 3.1 8B Instruct on ARC-C, GPQA and MMLU, while Llama 3.1 405B scores 85.2% on MMLU and 96.8% on GSM8K.

June 23, 2025Monday
  1. Vapi Blog22

    8 Alternatives to Azure for Voice AI STT on Vapi

    Vapi now offers eight speech-to-text providers — Gladia, AssemblyAI, Deepgram, Cartesia, Talkscriber, OpenAI, Speechmatics and ElevenLabs — as drop-in alternatives to Azure for voice agents. Switching providers takes a single JSON parameter change, letting developers benchmark latency, cost and compliance. Gladia claims sub-100 ms end-to-end latency, AssemblyAI bundles summaries and PII redaction in one API call, and Cartesia runs fully on-premises.

June 19, 2025Thursday
  1. Vapi Blog22

    Gemini Flash vs Pro: Comparing Google's Latest LLMs

    Google's Gemini 2.5 Flash and 2.5 Pro differ mainly in speed versus reasoning depth, both offering 1M-token context windows and native availability in Vapi for voice agents. Flash returns its first token in 0.21–0.37 seconds at 163 tokens per second and costs $0.15/$0.60 per million input/output tokens, while Pro costs $1.25/$10 and adds a Deep Think mode, with a 2M-token window planned.

June 18, 2025Wednesday
  1. Modal Blog58

    Modal shows how to run FLUX.1-dev three times faster

    Modal engineers describe making Black Forest Labs' FLUX.1-dev run about three times faster as an autoscaling service, cutting a 1024x1024 image from roughly 6.75 seconds to under three. Standard optimizations, namely the Torch compiler with max-autotune, fused QKV attention and channels-last memory layout, gave about 1.5x, while First Block Caching from the ParaAttention repo, based on the TEACache paper, added another 2x at a threshold of 0.12. Compile caches and Modal Memory Snapshots cut cold start latency by 30x.

June 17, 2025Tuesday
  1. Vapi Blog38

    Deepgram Nova-3 vs Nova-2: How the STT Engine Evolved

    Deepgram's Nova-3 cuts streaming word error rate from Nova-2's 8.4% to 6.84% and reaches 5.26% batch WER, a 54.3% reduction in streaming errors. Nova-3 replaces Nova-2's separate domain-specific models with a single adaptable network and adds real-time multilingual conversation handling, up to 100 self-serve custom terms without retraining, and redaction for up to 50 entities. Both models run natively on Vapi with identical endpoints, so switching is a parameter change, though Nova-3 sits at Deepgram's premium pricing tier.

June 9, 2025Monday
  1. Modal Blog40

    Modal 1.0 Released: More Stable Python SDK with Explicit Configuration

    Modal has launched version 1.0 of its Python SDK client, a milestone the company says brings a new level of maturity and stability to its serverless cloud compute platform, with far fewer breaking changes expected going forward. The release removes "magic" behaviors such as automatic mounting of local packages, splits core functionality across separate decorators like the new @modal.concurrent, and consolidates local file handling under the modal.Image class. Modal also plans SDKs for other languages and is exploring how the platform should serve AI-driven agent workflows.

June 4, 2025Wednesday
  1. Vapi Blog22

    Three Examples of Voice AI Improving Logistics

    Vapi outlines three voice AI use cases in logistics: ShipEX uses voice intelligence across 350 trucks to route maintenance alerts and accident reports, the Idaho State Liquor Division's voice picking system lifted pick rates 35%, and maintenance technicians log issues by voice to auto-create CMMS work orders. The article also details API integration with TMS, CRM, and WMS systems and deployment considerations.

  2. Vapi Blog22

    11 Great ElevenLabs Alternatives: Vapi-Native TTS Models

    Vapi now offers 11 built-in text-to-speech models as alternatives to ElevenLabs, including Neuphonic's Neu-hq and Neu-fast, Cartesia's Sonic lineup, Azure TTS, OpenAI TTS, Deepgram, Smallest AI's Lightning, LMNT, PlayHT, Hume's Octave and Rime AI's Mist v2. Developers can switch between them in minutes without re-coding, and Vapi says it handles over a million voice calls per day across these models.

May 30, 2025Friday
  1. Modal Blog42

    Modal adds Nvidia B200 and H200 GPUs at $6.25 and $4.54 per hour

    Modal now offers Nvidia B200 and H200 GPUs to any account holder with no sales contact or quota request, priced at $6.25/hour and $4.54/hour respectively. The B200 carries 180 GB of HBM3e memory and 8 TB/s bandwidth, and in Modal's vLLM test running DeepSeek V3 in 8-bit precision, 8xB200 cut median time-to-first-token 2.5x versus 8xH200 at 1 request per second.

May 28, 2025Wednesday
May 26, 2025Monday
  1. Vapi Blog10

    Introduction to SRTP: Secure Real-Time Transport Protocol

    SRTP secures RTP voice and video streams by adding encrypted payloads, authentication tags, and a master key identifier to each media packet. It relies primarily on AES counter mode and HMAC-SHA1 authentication tags, with sequence numbers blocking replay attacks, and pairs with DTLS-SRTP or SDES for key exchange. SRTP is positioned as a lower-overhead, application-layer option for real-time communications, with GDPR, HIPAA, and CPRA cited as compliance drivers.

  2. Vapi Blog18

    Env Files and Environment Variables for Voice AI Projects

    A guide on using .env files and environment configuration in voice AI projects, covering dotenv setup for Node.js and Python, adding .env to .gitignore, and Vapi-specific keys like VAPI_API_KEY and VAPI_PROJECT_ID. It cites GitGuardian's finding of 12.8 million secrets leaked on GitHub in 2023, a 28% increase, and recommends separate env files per environment plus tools such as git-crypt, SOPS, AWS Secrets Manager and HashiCorp Vault.

May 23, 2025Friday
  1. Vapi Blog18

    Amazon Lex vs Dialogflow: Complete Platform Comparison Guide

    Amazon Lex suits teams already on AWS needing tight integration with Amazon services, while Dialogflow offers broader language support and an easier learning curve for newcomers. Lex uses Alexa's technology with pay-as-you-go pricing including a free tier of 10,000 text and 5,000 speech requests monthly for the first year, and Dialogflow supports over 30 languages with pre-built agents and sentiment analysis.

  2. Vapi Blog8

    Mastering Environment Variables for Vapi Voice AI Integration

    Vapi's guide explains how to configure environment variables for its Voice AI integration, covering temporary vs. permanent and user-level vs. system-level variables. It details setup on Windows, macOS and Linux with Vapi-specific examples like VAPI_API_KEY and VAPI_MULTILINGUAL_SUPPORT, plus naming and security best practices such as never hardcoding API keys.

May 22, 2025Thursday
  1. Modal Blog58

    Modal launches Batch, processing 1 million jobs with one line of code

    Modal launched Modal Batch, a batch-processing interface backed by a rebuilt durable queue that can queue 1 million inputs at once, up from 2,000, and guarantees execution up to 7 days after a job is launched, up from 1 day. A new native handler, .spawn_map, spawns thousands of containers over a set of inputs without users managing async logic, and Harvey reports a 10x speed-up in its document-processing pipeline after fanning inputs across 1,000 containers. Suno uses it to run GPU-accelerated preprocessing on thousands of GPUs, and Achira relies on its retry mechanism for quantum mechanical datasets.

May 20, 2025Tuesday
  1. Modal Blog34

    Modal raises serverless KV store Dict limits to unlimited storage with new locking

    Modal upgraded its serverless Dict key-value store: new Dicts now have unlimited storage instead of the previous 10GiB cap, and item expiry changed to 7 days since last write or read. A new skip_if_exists flag on .put() enables distributed locking and "exactly once" semantics, and Dicts now offer data durability. The changes apply only to newly created Dicts.

May 7, 2025Wednesday
April 30, 2025Wednesday
  1. Modal Blog38

    Modal launches alpha SDKs for JavaScript and Go

    Modal released alpha versions of its official SDKs for JavaScript and Go, letting developers call deployed Modal Functions and create isolated sandboxes from Node, Next.js, or Go services without writing Python. The SDKs cannot deploy functions, which still require Python, and only support JSON-like or bytes data types for now. Modal says it can run 10,000 sandboxes in either language without issues.

April 24, 2025Thursday
  1. Modal Blog22

    How Lemon Slice built real-time generative video with Modal and Daily

    Lemon Slice built Lemon Slice Live, a product for real-time video conversations with AI characters, using Modal and Daily's low-latency primitives to reach an end-to-end video response latency of 3-6 seconds. The pipeline runs a Pipecat server and a 1B parameter video model on separate Modal containers that communicate via Modal's Tunnel and are co-located with region selection, with Deepgram, Grok and ElevenLabs handling speech-to-text, LLM response and text-to-speech. Earlier, two Python Modal Functions let Lemon Slice scale to 10,000 requests per hour without managing an ECS cluster, job queue or load balancer.

April 18, 2025Friday
  1. Modal Blog22

    How sync. uses Modal to lipsync 100 hours of video a day

    sync. is using Modal to lipsync over 100 hours of video a day, splitting long videos into scenes that run face detection and translation across n T4 GPU containers before passing each scene to its proprietary LipSync GenAI model on A100s. The team deploys up to 95 times a day and shipped 10 major model variants to production in one year, after moving off a Google Colab prototype and AWS Lambda.

April 17, 2025Thursday
  1. Modal Blog25

    Modal product updates: 1M-input batch jobs, ephemeral apps, and a TensorRT-LLM example

    Modal now lets each Function queue up to 1 million inputs, up from 2k, with higher .spawn() rate limits and FunctionCall results retained for 7 days. The update also adds ephemeral apps via app.run() inside containers, Git commit info for apps in the CLI and dashboard, and a new TensorRT-LLM example serving LLMs at under 400 ms latency.

April 1, 2025Tuesday
  1. Vapi Blog20

    Scaling Client Intake with Vapi Voice AI Agents: A Consultant's Case Study

    Consultant Christian Ulstrup of GSD at Work used Vapi voice AI agents to replace scheduled discovery calls with an asynchronous voice intake system for his AI Action Sessions workshops. Participants call a dedicated Vapi phone line at their convenience, and transcripts plus AI-generated summaries flow into Slack via Zapier and are logged in HubSpot. For one recent client, 20 participants across the globe contributed over 3 hours of discovery, and the session earned a 5 out of 5 rating.

March 17, 2025Monday
February 24, 2025Monday
  1. Modal Blog40

    A high-level guide to GPU utilization: allocation, kernel, and Model FLOP/s utilization

    Modal published a guide distinguishing three things commonly called "GPU utilization": GPU Allocation Utilization (GPU-seconds running application code over GPU-seconds paid for), GPU Kernel Utilization (fraction of time running code on GPUs), and Model FLOP/s Utilization (share of theoretical arithmetic bandwidth used). The guide focuses on neural network inference workloads, which Modal calls a revenue center rather than a cost center.

February 21, 2025Friday
  1. Modal Blog35

    Modal open sources GPU Glossary under CC BY 4.0

    Modal has open sourced its GPU Glossary, an interlinked compendium of short documents on GPU programming topics, on GitHub under a Creative Commons BY 4.0 License. The glossary was first released two months ago to fill a gap in cross-stack GPU reference material, and community requests to contribute prompted the move. The repo's GitHub Issues already list planned extensions, including two sample matmul kernels and a section on Thread Block Clusters.

February 14, 2025Friday
  1. Modal Blog22

    Modal product updates: Logs v2, live container profiling, and region selection on all plans

    Modal shipped a refreshed logs dashboard with time-range input filtering, live container profiling for debugging stuck containers or inputs, and Debug Shells that run commands like vim, ps, and strace inside running containers. Region selection is now available on all plans, priced with a multiplier on base prices, and the client adds OIDC authentication for cloud bucket mounts via the new oidc_auth_role_arn field.

January 28, 2025Tuesday
  1. Modal Blog62

    Modal adds memory snapshots to cut serverless cold starts by 2.5x

    Modal introduced memory snapshot restore for its serverless GPU container runtime, which it says makes cold starts about 2.5x faster than standard container startup. A Stable Diffusion inference Function that normally takes around 13 seconds now restores in 3.5 seconds, and a simple import torch program drops from roughly 5 seconds to about 1.05 seconds at p50 and 0.69 seconds at p0. Snapshots are built on gVisor's checkpoint/restore and are created on demand, with automatic fallback to a standard container startup when a restore fails; GPU state is not yet saved, so it must be rebuilt after restore.

    Why it matters: Modal explains how gVisor-based memory snapshots cut serverless cold starts, with concrete restore timings for a Stable Diffusion Function and a torch import example.

January 21, 2025Tuesday
  1. Modal Blog20

    Modal product updates: L40S GPUs, proxy auth tokens, and sandbox disk snapshotting

    Modal now offers NVIDIA L40S GPUs at $1.95/hr, with 48GB of memory, up to 40% faster performance on memory-bound tasks and over 100% speedup on compute-bound jobs versus its A10 GPUs. The platform also added Proxy Auth tokens for gating web endpoints, a Filesystem API for Sandboxes, and disk snapshotting that lets users branch new Sandbox instances from snapshots to eliminate cold starts. Modal additionally announced completion of its SOC 2 Type 2 certification.

  2. Modal Blog58

    Modal Sandboxes 正式可用，为 LLM 与用户代码提供隔离执行环境

    Modal announced the general availability of Sandboxes, its primitive for safely running untrusted code from LLMs, users, or third-party sources in an isolated environment separate from other Modal resources. Sandboxes offer a simple exec API, runtime-configurable environments via the Image API, and filesystem snapshots for persistence. Modal also upstreamed support into SWE-bench, where a --modal flag lets the Verified benchmark's 500 tasks run in about 7 minutes, and cited Quora's Poe, Codegen, and Relevance AI as customers.

December 28, 2024Saturday
  1. Modal Blog22

    Modal product updates: faster memory snapshotting, OIDC, async job queues

    Modal shipped a product update round led by a major upgrade to its beta memory snapshotting feature, which it says can cut cold start times by half for models like Stable Diffusion. The platform also added OpenID Connect support for short-lived identity tokens, eStargz image compression for faster builds from registries like ECR and Docker Hub, and is expanding async job queues toward a 1 million input limit, up from 2k.

December 19, 2024Thursday
  1. Modal Blog25

    Modal Adds NVIDIA L40S GPUs for AI Inference at $1.95/hr

    Modal now offers NVIDIA L40S GPUs to all users at $1.95/hr, positioned as a faster, larger-memory alternative to its most popular inference accelerator, the A10. The L40S doubles GPU RAM to 48GB, letting models like Flux.1-schnell run in 16bit precision on a single GPU, and delivers roughly 40% speedup on memory-bound jobs and over 100% on compute-bound 16bit Tensor Core jobs.

December 10, 2024Tuesday
  1. Modal Blog30

    What is LLM fine-tuning? A Guide to Techniques and Tools

    Fine-tuning adjusts pretrained LLM weights to fit a specific task or domain, delivering higher quality than prompt engineering at lower cost and latency; a fine-tuned Llama 7B can be roughly 50 times more cost-effective per token than GPT-3.5 with comparable performance. The guide covers use cases such as structured output, style adherence and domain-specific instruction, plus options including OpenAI, Predibase, Modal, Google Colab and AWS SageMaker, and frameworks HuggingFace transformers, TRL and Axolotl.