Skip to content
Only First-party#Tutorials / practice
April 1, 2025Tuesday
  1. Vapi Blog20

    Scaling Client Intake with Vapi Voice AI Agents: A Consultant's Case Study

    Consultant Christian Ulstrup of GSD at Work used Vapi voice AI agents to replace scheduled discovery calls with an asynchronous voice intake system for his AI Action Sessions workshops. Participants call a dedicated Vapi phone line at their convenience, and transcripts plus AI-generated summaries flow into Slack via Zapier and are logged in HubSpot. For one recent client, 20 participants across the globe contributed over 3 hours of discovery, and the session earned a 5 out of 5 rating.

February 24, 2025Monday
  1. Modal Blog40

    A high-level guide to GPU utilization: allocation, kernel, and Model FLOP/s utilization

    Modal published a guide distinguishing three things commonly called "GPU utilization": GPU Allocation Utilization (GPU-seconds running application code over GPU-seconds paid for), GPU Kernel Utilization (fraction of time running code on GPUs), and Model FLOP/s Utilization (share of theoretical arithmetic bandwidth used). The guide focuses on neural network inference workloads, which Modal calls a revenue center rather than a cost center.

December 10, 2024Tuesday
  1. Modal Blog30

    What is LLM fine-tuning? A Guide to Techniques and Tools

    Fine-tuning adjusts pretrained LLM weights to fit a specific task or domain, delivering higher quality than prompt engineering at lower cost and latency; a fine-tuned Llama 7B can be roughly 50 times more cost-effective per token than GPT-3.5 with comparable performance. The guide covers use cases such as structured output, style adherence and domain-specific instruction, plus options including OpenAI, Predibase, Modal, Google Colab and AWS SageMaker, and frameworks HuggingFace transformers, TRL and Axolotl.

November 20, 2024Wednesday
September 24, 2024Tuesday
September 18, 2024Wednesday
August 5, 2024Monday
  1. Modal Blog62

    Modal replicates Large Language Monkeys: LLaMA 3.1 8B matches GPT-4o on HumanEval via search

    Modal's Charles Frye replicated the core claim of the Large Language Monkeys paper on a new dataset, showing that repeated sampling lifts LLaMA 3.1 8B from 66.4% pass@1 to 90.5% pass@100 and 95.1% pass@1000 on HumanEval, versus GPT-4o's reported 90.2% pass@1. The runs used vLLM on Modal serverless GPUs and Modal Sandboxes for parallel test execution, cost well under $50, and showed smooth, predictable scaling across three orders of magnitude.

    Why it matters: A hands-on replication shows how repeated sampling with a small open model can approach GPT-4o on HumanEval, offering a concrete, low-cost recipe for trading compute for accuracy.

June 20, 2024Thursday
  1. Modal Blog25

    Run GPU jobs from Airflow with Modal

    Modal can now be triggered directly from an Airflow DAG, letting teams run GPU-backed AI/ML tasks without setting up Celery or Kubernetes executors. The guide covers installing Modal in an Airflow environment, setting MODAL_TOKEN_ID and MODAL_TOKEN_SECRET, and two approaches: calling deployed Modal Functions via lookup, or building a custom operator that runs Python in Modal Sandboxes. A Modal Airflow provider package is in progress.

June 6, 2024Thursday
  1. Modal Blog38

    How Modal catches crypto miners using syscall signatures

    Modal added a runtime program-analysis component called seccheck to its serverless container runtime, detecting and disabling cryptomining Functions before banning the offending users. Because miners can change metadata but not their programs, Modal analyzes syscall logs rather than source code, binaries, or container images. The system avoids ptrace-based strace overhead, which can slow syscall-heavy programs by over 100x.

May 23, 2024Thursday
  1. Modal Blog22

    How Hunch Uses Modal Sandboxes to Run AI-Generated Code Safely

    Hunch, a spatial canvas that combines models such as OpenAI GPT-4o, Anthropic Claude 3 Opus and Stable Diffusion 3, uses Modal Sandboxes to execute AI-generated code safely in isolated, ephemeral containers. The integration required only a few lines of Python and supports pip installs, attached storage volumes and automatic scaling. Hunch said the setup lets non-coders build workflows that fetch API data or send results to Slack webhooks.

May 21, 2024Tuesday
  1. Modal Blog31

    Fine-tuning Stable Diffusion on Heroicons to Create an Infinite Icon Library

    Modal fine-tuned a Stable Diffusion model on the roughly 300-SVG Heroicons library to generate new black-and-white line icons in that style, including ones for concepts like barack-obama and golden-retriever. The full pipeline — downloading the pretrained model, full fine-tuning with HuggingFace Diffusers' train_text_to_image.py script, and serving a Gradio web UI — runs on Modal's serverless GPU platform. The write-up walks through dataset prep (SVG-to-PNG conversion, white backgrounds, caption metadata.csv) and notes that full fine-tuning worked best, though LoRA-style parallel adapters can also work for small datasets.

April 26, 2024Friday
  1. Modal Blog50

    Fine-tuning an open source embedding model to beat OpenAI's text-embedding-3-small

    A fine-tuned open source embedding model can outperform OpenAI's text-embedding-3-small in accuracy at lower cost using only a handful of examples. Modal ran an experiment on the Quora duplicate-questions dataset to find how much fine-tuning data a simple question-answering use case needs, and describes how to build one on its serverless infrastructure.

March 26, 2024Tuesday
  1. Modal Blog22

    How Ramp automated receipt processing with fine-tuned LLMs on Modal

    Ramp used Modal to fine-tune its LLMs and build a text-to-structured-JSON model for receipt management, cutting receipts requiring manual intervention by 34% on infrastructure estimated 79% cheaper than major LLM providers like OpenAI. Modal functions trained many candidate models in parallel, persisted weights to Modal volumes, and served a parametrized inference endpoint. One engineer also ported a PII-stripping script for 25,000 invoices to Modal, parallelizing it on 256 cloud workers to finish in 20 minutes for $100 instead of 3 days locally.

March 14, 2024Thursday
  1. Modal Blog36

    Modal's modal-http: Serving Web Functions on Serverless Infrastructure

    Modal built modal-http, an HTTP and WebSocket stack that lets its serverless functions accept web requests, with full WebSocket support recently completed. Unlike AWS Lambda's 15-minute and 50 MB limits, Modal containers can use up to 64 CPUs, 336 GB of memory and 8 Nvidia H100 GPUs, so the team translates HTTP into function calls rather than fronting containers with a conventional reverse proxy.

January 23, 2024Tuesday
  1. Modal Blog40

    Embedding English Wikipedia in under 15 minutes with Modal and Hugging Face TEI

    Modal and Hugging Face's Text Embedding Inference server can embed the entire English Wikipedia in about 15 minutes for just over $15, using Modal's serverless GPU scaling. The post argues open-source embedding models beat API-based ones like OpenAI's text-embedding-ada-002 for production RAG, since they can be fine-tuned on user data and avoid rate limits and per-token costs.

July 21, 2023Friday
  1. Cursor Changelog62

    Cursor analyzes Llama-2-70B inference cost and latency against GPT-3.5

    Cursor measured Llama-2-70B inference on 2 80-GB A100 GPUs and found it over 3x cheaper than GPT-3.5 for prompt tokens but far more expensive for generated tokens at small batch sizes. Prompt processing costs about $0.00042 per 1K tokens versus GPT-3.5's $0.0015, while generation reaches $0.066 per 1K tokens at batch size 1 and only approaches GPT-3.5 pricing at large batches that push time to first token to several seconds. Cursor recommends open-source models for prompt-heavy tasks like classification and reranking, and closed-source APIs for generation-heavy workloads.

    Why it matters: Cursor's measured cost and latency data shows when self-hosting Llama-2-70B beats GPT-3.5 and when it does not, offering a reusable framework for open-source model serving decisions.