Skip to content
Only First-party#Deployment / engineering
July 29Wednesday
  1. Berkeley AI Research58

    From CUDA to MLX: K-Search Brings Kernel Optimization Expertise to Apple Silicon

    IBM Research extended K-Search, the evolutionary kernel search framework from UC Berkeley Sky Lab, with an MLX backend and a structured CUDA-to-MLX translation layer that adapts existing CUDA kernels into Apple Silicon kernels. On an MLX attention kernel the approach reached 0.97x the speed of Apple's native MLX Attention kernel, up from 0.26x without the translation context, and an evolved Mamba SSM kernel reached up to 20x prefill speedup over the community mlx-lm implementation on an M1 Max. The authors attribute the gain mainly to the quality of context and constraints given to the LLM rather than its ability to write Metal code, and say the method is not specific to MLX.

July 27Monday
  1. Modal Blog75

    Moonshot's Kimi K3 now available on Modal with day-zero vLLM support

    Modal made Moonshot's Kimi K3 available on its platform on release day, offering the 2.8 trillion parameter multimodal model with a 1M token context window through a token-priced Shared API and dedicated Auto Endpoints. Modal says a custom DFlash speculator tuned to K3's architecture raises interactivity from 100 to 460 tokens per second and throughput from 800k to 1.5 million TPM per GPU on agentic workloads. The model uses a mixture-of-experts design with 16 of 896 experts active per token, and Moonshot contributed a new prefix-caching implementation to vLLM ahead of the release.

    Why it matters: Modal details how it brought Moonshot's 2.8T-parameter Kimi K3 to its platform on release day, including a custom DFlash speculator that lifts interactivity from 100 to 460 tokens per second.

July 24Friday
  1. Vapi Blog22

    How to reduce customer wait times with AI voice agents

    AI voice agents cut customer wait times in two ways: removing work from the queue through self-service deflection, intelligent routing, agent assist, and demand prediction, and answering instantly with no hold. SQM Group found customers abandon calls at 2 minutes 36 seconds on average, while ContactBabel put the UK mean abandonment rate at 8.4% in 2024. Vapi says a first voice agent can be stood up from its dashboard in minutes, then measured on ASA, abandonment, and first-call resolution.

July 21Tuesday
  2. Modal Blog58

    Modal and Cognition launch Devin Outposts with modal-devin sandbox integration

    Modal is a launch partner for Devin Outposts, a Cognition feature that moves Devin's code execution into an environment the customer controls while Devin's reasoning stays in Cognition's cloud. The open-source modal-devin library and CLI runs each Outpost session in a Modal Sandbox, giving access to GPU-backed machines, custom images, workspace snapshotting and scale-to-zero, and it is live in alpha starting July 21, 2026. Setup uses uvx modal-devin init, which generates an orchestrator function and a worker session function that teams can keep customizing.

July 16Thursday
  1. Modal Blog60

    Modal rebuilds its sandbox platform to run 1 million concurrent sandboxes, created in under a minute

    Modal has rebuilt its core sandbox platform from the ground up, and on the new system it ran 1 million sandboxes concurrently, creating all of them in under a minute. The redesign removes central coordination and any data store from the sandbox creation path: horizontally scaled scheduling servers pick a worker with an in-memory algorithm and call it directly, while workers publish state asynchronously to a Redis stream. Modal says sandbox start latency is under half a second at the median, and the new system is already available in Beta with a one-line code change to opt in.

    Why it matters: Modal rebuilt its sandbox control plane to drop central coordination, showing how far container scheduling can scale once global consistency leaves the critical path.

July 15Wednesday
  1. Modal Blog60

    Thinking Machines' Inkling multimodal model now available on Modal

    Thinking Machines released Inkling, a general-purpose multimodal model that takes text, image and audio inputs and outputs text, and Modal is offering it from day zero as a Managed Endpoint with token-based pricing. Inkling is a mixture-of-experts transformer with 975B total parameters, 41B active, a 1M token context window and native audio and vision, using five sliding-window attention layers for every full-attention layer. On agentic workloads with 8x B200s, Modal's custom DFlash speculator reached 250 tokens per second per user at 2.5M TPM of per-GPU throughput, 67% faster interactivity than the model's built-in speculative path at matched throughput.

    Why it matters: Modal's day-zero endpoint for Thinking Machines' 975B-parameter Inkling shows how a custom DFlash speculator lifts interactivity 67% over the model's built-in speculative path.

July 9Thursday
  1. Modal Blog22

    Modal Is a Computer: Recasting the Cloud Platform as a Machine

    Modal argues it should be understood as a computer rather than an "Uber for compute," a neocloud, or an inference provider, since it sells processor-seconds and byte-seconds rather than tokens. The post maps a conventional machine's ALU, memory hierarchy, I/O, NIC, and operating system onto Modal's containers, object storage, Input/Output Plane, Routing Plane, and Tunnels. It also aligns Modal Functions, Sandboxes, and Servers with stateless tools, Linux processes, and HTTP listeners.

July 7Tuesday
  1. Berkeley AI Research36

    Intelligence Is Free, Now What? Data Systems for, of, and by Agents

    UC Berkeley's Aditya G. Parameswaran and collaborators argue that as inference costs fall from roughly $30 per million tokens in early 2023 to under $1 today, data systems must be redesigned around three challenges: systems for agents, of agents, and by agents. They note that in a text-to-SQL benchmark with multiple agents per task, only 10-20% of sub-plans are distinct, so 80-90% of sub-queries duplicate work, and propose reuse, approximation, and proactive data systems.

July 6Monday
  1. Modal Blog38

    How to price serverless GPUs: Modal's interactive cost model

    Modal published an interactive tool for estimating serverless versus reserved GPU costs, defaulting to current rates for a B200 GPU serverlessly on Modal and via three-year hyperscaler reservation. The model shows that when an application's peak-to-average ratio exceeds the reservation discount rate, serverless GPUs cost less overall, and Modal cites observed discounts of 2-5x against peak-to-average ratios of 5-10x.

July 1Wednesday
  1. Modal Blog58

    Modal Research and HeavyBall's MRP brings multi-token prediction to diffusion LMs

    Modal Research and NYU Shanghai's HeavyBall Research introduce Multi-token Residual Prediction (MRP), a 3-layer module attached to a frozen diffusion LM backbone that predicts the inter-step logit residual instead of the full next-step distribution. On SDAR-4B GSM8K, MRP holds 70.9% accuracy at K=3 versus 5.9% for naive MTP distillation, and in SGLang it reaches up to 1.56x throughput losslessly in speculative mode or recovers up to +16 accuracy points under aggressive low-threshold dynamic decoding. The paper, code, SGLang implementation, and SDAR-MRP models on Hugging Face are released.

June 30Tuesday
  1. Modal Blog66

    Anthropic launches Claude Science with Modal as scalable compute backend

    Anthropic launched Claude Science on June 30, an AI workbench where life sciences researchers run computational workloads directly inside a conversation with Claude. Modal integrates as the compute backend: workloads needing GPUs or many concurrent CPUs route automatically to Modal sandboxes from a researcher's connected workspace, while default execution stays sandboxed on the local machine. Modal cites fan-out virtual screening across hundreds of containers, per-step GPU access for models like ESMFold2, shared Modal Volumes storage and reproducible Modal Images, and commits up to $100,000 in compute for Anthropic's AI for Science Claude Science Cohort at $500–$2,000 per project, with applications open through July 15, 2026.

    Why it matters: Modal's post explains how Claude Science routes GPU and parallel workloads to Modal sandboxes, with concrete life-science examples and a compute grant for academic labs.

June 25Thursday
  1. Modal Blog55

    Modal launches serverless Servers for low-latency HTTP, WebSocket and gRPC traffic

    Modal launched Servers, a regionalized autoscaling pool of HTTP server replicas for HTTP, WebSocket and gRPC traffic aimed at latency-sensitive workloads such as LLM inference for interactive agents. Modal says it cut p50 latency from 39ms to 6ms by keeping control-plane lookups and queueing off the request path, so clients get a 503 when no replica is available and must handle load shedding in the application. The routing layer runs on AWS EKS with Envoy at the edge and Modal's in-house Rust proxy fprs, built on Cloudflare's pingora, handling domain association and load balancing, with config read from Spanner and cached in memory.

June 24Wednesday
  1. Vapi Blog15

    Vapi Answers Top Questions from Its SVG Lead Qualification Webinar

    Vapi published answers to the seven most-asked questions from its webinar with Spring Venture Group, where Homer Kay described how SVG uses Vapi voice agents to qualify Medicare leads before transferring callers to licensed agents. The answers cover outbound calling compliance, API-first integrations, swapping STT, LLM and TTS components independently, human handoff, and HIPAA-eligible configurations with Zero Data Retention.

  2. Modal Blog58

    Modal launches Auto Endpoints, citing speculative decoding for state-of-the-art inference latency

    Modal is launching Modal Auto Endpoints, which bring Modal's infrastructure scalability to state-of-the-art inference performance with a single click while keeping control of the code. Modal says the main latency win comes from speculative decoding with custom DFlash speculator models, which cut about 100ms of end-to-end latency in a Decagon Voice case study, roughly 40% of server-side decode latency, making it over 60ms faster than the fastest proprietary alternative. Modal also released open source DFlash speculator models with Z Lab and SGLang, including one for Qwen 3.5 397B-A17B that improves on MTP by over 50%.

June 23Tuesday
  1. Modal Blog60

    Modal launches Auto Endpoints for self-serve, OpenAI API-compatible LLM inference

    Modal launched Auto Endpoints, a self-serve service that deploys production-grade LLM inference as an OpenAI API-compatible endpoint backed by a Modal App the user can see and control. Each supported model ships with a starting deployment tuned from Modal's own benchmarking, using engines like SGLang and the DFlash speculative decoding drafter from Z Lab, and users can inspect GPU choices, engine flags and inference metrics such as TTFT, ITL and speculative decoding acceptance length. Modal also released Modal Servers from beta for regionalized routing with 5ms overhead, and said it generates endpoint deployments with an internal agentic system still monitored by human engineers.

    Why it matters: Modal's Auto Endpoints expose the serving stack, engine metrics and benchmark recipes that managed inference providers keep hidden, giving teams a self-serve path to owning open-model inference.

June 22Monday
  1. Modal Blog36

    Modal explains why sandbox "started" ≠ "ready" and how warm pools cut perceived startup latency

    Modal argues that container scheduling and boot time, which most sandbox benchmarks measure as time to interactive, is the smallest part of what users actually wait through; application-level setup after the container starts often takes multiple seconds. Modal is making Readiness Probes generally available to signal when a sandbox has finished initializing, and recommends warm sandbox pools built on modal.Queue plus Directory Snapshots to move setup off the user's critical path.

June 19Friday
  1. Modal Blog60

    Modal releases DFlash speculators for Qwen 3.5 and 3.6, claiming 5-20% extra speedup

    Modal released a set of DFlash draft models for the Qwen series on Hugging Face, including Qwen 3.6 35B-A3B-DFlash and Qwen 3.5 variants from 4B to 122B-A10B, plus a DFlash speculator for Qwen 3.5 397B-A17B built with SGLang. The company says the new drafters add 5-20% speedup over existing DFlash baselines and can run Qwen 3.5 122B-A10B at over 1000 tok/s at concurrency 1 on a B200 node, versus 250 tok/s without speculation. The post argues speculative decoding delivers 2x-3x gains where kernel optimization yields small percentage points, and presents a roofline model for estimating speedup from acceptance length.

    Why it matters: Modal explains why it bets on speculative decoding and releases DFlash draft models for the Qwen series, with measured speedups and a roofline model for estimating gains.

June 17Wednesday
  1. Vapi Blog40

    How Vapi Bootstrapped 15 Homepage Voice Agents in Code

    Vapi built the 15 voice agents on its homepage — 3 use cases across 5 languages — by generating them from a single ROUGH_DRAFT.md file instead of hand-editing each in the dashboard. The four-phase framework (plan, draft, bootstrap, harden) scaffolds a TypeScript project with OpenAI gpt-4.1 at temperature 0.5, ElevenLabs and Deepgram/Soniox voice stacks, and an idempotent upsert loop that stores assistant IDs in .env.local. The repo is public: clone it, add VAPI_PRIVATE_KEY, and run bun run bootstrap.

June 15Monday
  1. Modal Blog34

    Modal Product Updates: VM Sandboxes, Regional Routing, RBAC and More

    Modal shipped a batch of platform updates, including alpha VM Sandboxes that run a real Linux kernel to support Docker containers, eBPF, systemd and cgroups. Functions can now route through US West, EU West or Asia Pacific South instead of the default US East region, and RBAC is available to all Team and Enterprise plans. Sandbox readiness probes are now GA, and the Python SDK 1.5.0 and JS/Go SDK 0.8.0 set a 30-day default snapshot TTL.

June 11Thursday
  1. Modal Blog34

    Modal Makes FlashAttention-4 Faster for LLM Inference

    Modal contributed optimizations to the FlashAttention-4 kernel to speed up decode-heavy LLM inference workloads, where memory bandwidth is the bottleneck. The changes include porting the "split KV" parallelism technique to FA4 and replacing TMA cp.async.bulk loads with cp.async for irregular memory accesses. An FP8 input PR (merged April 17, 2026) delivers up to 1.16x throughput over the bf16 baseline and enables smaller KV caches.

  2. Factory62

    Factory rolls out automated security review in Droid

    Factory is rolling out automated security review in Droid, which runs a STRIDE-based security review alongside standard code review on every non-draft PR and posts findings with severity, a CWE reference, an explanation and a suggested fix as inline comments on the diff. The company says Droid's security review has been run across production codebases and surfaced findings that went through responsible disclosure, including CVE-2026-42876 in external-secrets and workos/workos-node#1578. Droid builds a lightweight threat model of the changed surface, scans for vulnerabilities such as OWASP Top 10 and OWASP LLM Top 10 issues, then validates each candidate finding against the diff to filter false positives before posting one deduplicated summary per PR.

    Why it matters: Factory's Droid now runs a STRIDE-based security review on every non-draft PR, and the post shows real CVEs it surfaced in production codebases.

June 1Monday
  1. Modal Blog46

    Modal: Reinforcement Learning for LLM Post-Training Is an Infrastructure Problem

    Modal says the bottleneck in RL post-training of LLMs is now infrastructure, not algorithms, and has released an open-source library drawn from running RL at scale. It details three pain points — glue code, cluster queueing and GPU under-utilization — and reports RDMA weight sync cutting Qwen3 8B update time from 2.62s to 41ms.

May 27Wednesday
  1. MiniMax Blog72

    MiniMax upgrades Agent as Mavis and launches Agent Teams for long-running tasks

    MiniMax upgraded MiniMax Agent under the new name Mavis and launched Agent Teams, which let multiple role-specific Agents run in parallel on the desktop for long, complex tasks. The system uses Leader, Worker and Verifier roles with a state-machine Team Engine that cycles tasks through producing, verifying and done, and it merges TokenPlan and Agent Plan into one subscription spanning CLI, API and Agent across M2.7, music, video and voice. MiniMax says the Agent will be open-sourced soon, expected alongside the MiniMax M3 model, while the desktop app is already available.

    Why it matters: MiniMax explains the design tradeoffs behind its Leader-Worker-Verifier Agent Team, including handoff, sharing and aggregation costs, offering a concrete reference for building multi-Agent runtimes.

  2. Modal Blog38

    Modal opens Role-Based Access Control to all Team and Enterprise users

    Modal is making Role-Based Access Control (RBAC) available to all users on Team and Enterprise plans, built around Environments as enforceable security boundaries. Restricted Environments make workspace members read-only by default, with only explicitly granted users or agents able to deploy and manage resources, and RBAC ties into Audit Logs for a full record of who did what and when.

May 21Thursday
  1. Vapi Blog20

    Vapi: Bringing Voice AI to ACCESS Chronic Care Panels

    Vapi argues that voice AI can now make voice the economically viable channel for ACCESS chronic disease management, citing Hiya data that 42% of consumers prefer voice for healthcare communication, up from 33% the prior year. The post details three production workflows, including Spring Venture Group's Medicare lead qualification agent, built by four engineers in one week, which lifted transfer-to-close rate by 3% and delivered over $5.8M in combined savings and new revenue.

May 20Wednesday
May 19Tuesday
  1. Modal Blog55

    Modal launches first-class Sandboxes integration with Claude Managed Agents

    Modal announced a first-class integration between Modal Sandboxes and Anthropic's Claude Managed Agents, letting all tool calls run in a self-hosted Modal Sandbox while Anthropic's managed agent loop handles session state. Modal cites fast cold starts on custom images, snapshot and volume persistence, burst pricing with no idle costs, short-lived connect tokens, and upwards of 100,000 concurrent sandboxes per customer. The company also published two reference examples, a CLI agent and a Slack bot, on the same Modal plus Managed Agents backbone.

May 12Tuesday
  1. Modal Blog58

    How Modal achieved truly serverless GPUs

    Modal published an engineering account of how it cut AI inference replica spin-up from roughly 2,000 seconds to about 50 seconds using four components: a buffer of idle healthy GPUs, a lazy content-addressed filesystem called ImageFS, host-side checkpoint/restore via gVisor's runsc, and CUDA checkpoint/restore built on Nvidia driver support. Across over ten thousand cold starts, mean vLLM boot latency fell from 95,679 ms to 13,797 ms and SGLang from 83,713 ms to 17,486 ms for a ~1 GiB Qwen 3 0.6B model in bf16. The post also notes limits, including single-GPU-only snapshots and host-environment sensitivity that requires multiple snapshots per deployment.

May 8Friday
  1. Berkeley AI Research38

    Adaptive Parallel Reasoning: A Survey of Efficient Inference Scaling

    A Berkeley AI Research survey examines adaptive parallel reasoning, in which a reasoning model itself decides when to decompose subtasks, how many concurrent threads to spawn, and how to coordinate them. The post contrasts this with fixed parallelism methods such as self-consistency, Best-of-N, Tree of Thoughts, MCTS, ParaThinker, GroupThink and Hogwild! Inference, which impose the parallel structure externally. Sequential reasoning scales linearly in tokens and latency and risks context-rot, motivating parallel threads that explore independently and concurrently.

May 4Monday
May 1Friday
  1. Vapi Blog34

    Open Stack vs Integrated Solution: Key Considerations for Voice AI Builders

    Vapi Blog interviews Cartesia's Iz Shalom on choosing between integrated and open modular voice AI stacks. Integrated stacks suit narrow use cases and fast proofs of concept, while open stacks fit multiple use cases, multilingual support, and swapping components. The piece urges teams to measure P99 latency in their own environment and build tight evaluations before production.

April 29Wednesday
  1. Modal Blog33

    Building an RL Theorem-Proving Workflow on Modal

    AE Studio details how it trained LLMs to prove math theorems with reinforcement learning on Modal, comparing Evolution Strategies against GRPO while keeping the training loop fixed. The setup runs vLLM inference on GPUs, Lean compiler verification in isolated CPU sandboxes, and a lightweight orchestrator, using per-function images, .map() fan-out, and Volumes for base weights.

April 23Thursday
  1. Factory55

    Factory launches Automated QA skill that tests apps like a real user and reports in PRs

    Factory released an Automated QA skill that drives apps the way a real user would, filling forms, typing into terminals and calling endpoints, then posts a structured report with screenshots, terminal snapshots and API traces as a single comment on each pull request. The skill runs on every push via Droid, can be configured as an optional CI check triggered by a PR label, comment command or manual dispatch, and is also available locally through /qa in any Droid session. Setup uses /install-qa, and the feature is available today in all Factory plans.

April 22Wednesday
  1. Replit Blog22

    Replit Wins 2026 Google Cloud AI Tooling Partner of the Year Award

    Google Cloud named Replit its 2026 AI Tooling Partner of the Year, recognizing the platform now used by more than 50 million people across 85% of the Fortune 500. The award follows the launch of Replit's Agent 4, which the company says is 10x faster than Agent 3 and runs multiple agents in parallel. Replit is available on the Google Cloud Marketplace, integrated with Cloud Run, Kubernetes Engine, and BigQuery.

April 20Monday
  1. Berkeley AI Research38

    GRASP: Gradient-Based Planning for World Models at Longer Horizons

    GRASP is a new gradient-based planner for learned world models that makes long-horizon planning practical by lifting trajectories into virtual states for parallel-in-time optimization, adding stochasticity to state iterates for exploration, and reshaping gradients so actions receive clean signals while avoiding brittle state-input gradients through high-dimensional vision models. The work addresses ill-conditioned computation graphs, non-greedy local minima, and high-dimensional latent failure modes that make long-horizon planning fragile. It was done with Mike Rabbat, Aditi Krishnapriyan, Yann LeCun, and Amir Bar.

April 17Friday
  1. OpenAI Developers Blog60

    OpenAI publishes API deployment checklist for GPT-6 workloads

    OpenAI published an API deployment checklist that walks developers through moving workloads onto the Responses API and the GPT-6 model family, including gpt-6-astra, gpt-6.1-sol, gpt-6 Sol and gpt-6 Luna. The guide covers reasoning.effort levels from low to max, text.verbosity, tool_search with deferred tool loading, Programmatic Tool Calling, Multi-agent, compaction, prompt caching and WebSocket mode, which OpenAI says has shown up to roughly 40% faster end-to-end execution for workflows with 20 or more tool calls. It also details reliability and safety steps such as handling 429 and 503 errors, background=True for long jobs, safety_identifier and misalignment monitoring.

    Why it matters: OpenAI's official deployment checklist consolidates model selection, reasoning effort, tool search, compaction and WebSocket mode into one migration guide for GPT-6 workloads.

April 15Wednesday
  1. Vapi Blog50

    Vapi Launches Vapi Monitoring for Production Voice Agents

    Vapi released Vapi Monitoring, a monitoring product that lets teams create customizable monitors for voice agents and get alerted when thresholds are hit. It covers effectiveness and compliance monitoring for Enterprise subscriptions, plus technical and infrastructure monitoring, and surfaces issues with affected call counts and recommended fixes.

  2. Modal Blog58

    Modal and OpenAI Agents SDK: building a parallel coding agent harness with sandboxes and GPUs

    Modal published a walkthrough on building a custom coding agent harness on top of the OpenAI Agents SDK, which OpenAI launched today, using Modal Sandboxes to give agents isolated Linux environments and GPUs. The guide starts from a minimal exec-based agent, then adds sessions for memory, an orchestrator plus subagent pool for parallel async work, GPU quotas, filesystem snapshots to deduplicate setup, and a skills subsystem, ending with a harness that runs OpenAI's Parameter Golf challenge across parallel GPU subagents. Modal cites Ramp's fleet of background coding agents on Modal, which it says now produces more than half of the company's PRs.

April 14Tuesday
  1. Vapi Blog27

    Vapi Engineers Share Lessons on Deploying Voice AI at Enterprise Scale

    Vapi's Forward-Deployed Engineers say the demo-to-production gap is where most enterprise voice AI initiatives stall, requiring extensive testing, guardrails and intent-based design rather than exact phrase matching. The team warns of "personality drift," where agents adopt callers' speech styles, and notes LLMs are non-deterministic, so deployments need defined error tolerance and continuous iteration. They advise starting with cost optimization but targeting revenue generation, and benchmark a good interaction at under a minute.