Skip to content
#Data / training
October 2Friday
  1. Meta AI Research Blog71

    Meta shares six math papers co-authored with mathematicians and Muse Spark

    Meta published six mathematics papers produced in collaboration with mathematicians who used Muse Spark 1.1 and 1.2 in Thinking Mode through the regular meta.ai chat interface, with no custom research scaffold. Five of the papers answer previously open questions, covering Gaussian ellipsoid fitting thresholds, finite-time blow-up for a biharmonic nonlinear Schrödinger equation, a 384-element counterexample to a group-theory conjecture, tightness of a cycle-based relaxation in binary polynomial optimization, and a link between string two-point functions and height functions on curves. Each paper marks which passages were primarily drafted by researchers and which by AI, credits prior work, and acknowledges independent concurrent solutions from other teams.

    Why it matters: Meta 公布六篇由数学家与 Muse Spark 协作完成的论文，其中五篇给出此前开放问题的答案，并标注 AI 与人类各自撰写的段落。

October 1Thursday
  1. Zyphra Research38

    How Hybrid NoPE Models Learn Position Without Position Encodings

    Zyphra Research explains how hybrid language models recover positional information even when global attention layers use no explicit position encoding. Local layers such as sliding window attention and gated linear attention create a recency bias in the residual stream, and global NoPE attention learns query and key projections that read it out as an implicit relative position encoding. The findings offer a mechanistic account of why interleaving local mixing with global NoPE attention works well and point to positional representations that extrapolate beyond training sequence lengths.

  2. Goodfire Research58

    Goodfire develops protein-embedding biosecurity monitors for AI agents

    Goodfire Research developed sequence-aware biosecurity monitors that use protein language model embeddings to flag concerning DNA or protein sequences in agent workflows. On a custom dual-use benchmark built around protein toxins, the monitors outperformed frontier model safeguards with fewer refusals on benign requests, reached roughly 0.98 AUROC on paraphrases whose parent proteins were in the screening database, and ran in milliseconds per sequence. The authors report that detection improves as protein models get more capable, and note the paraphrase results rely on in-silico estimates and do not establish whether redesigned proteins retain biological activity.

September 7Monday
  1. Charlie O'Neill38

    Let's Bet on RL Environments: Why Useful Data and Environments Will Win

    An argument that people who claim everything will become RL environments are not acting on that belief, and that useful RL environments will grow exponentially scarcer and more valuable as models approach RSI. The piece urges building long-horizon tasks in domains of real expertise rather than Goodhart-style task farms, and argues a broad spread of environments beats a small set of frontier-hard ones.

September 4Friday
  1. Modal Blog22

    Markov Robotics Uses Modal to Run End-to-End Robot Training

    Markov Robotics is training a world model for zero-shot general-purpose robot dexterity, using fewer than ten hours of robot data where comparable results typically need thousands of hours. The company runs nearly all its compute on Modal, from preprocessing hundreds of terabytes of video to 2–10 concurrent training runs and remote inference via Modal Tunnels at roughly 20 milliseconds round-trip latency. Preprocessing that could take a full day completes in twenty to thirty minutes, with fan-out to upwards of 500 GPUs at once.

September 2Wednesday
  1. Charlie O'Neill38

    Trying to Actually Define Continual Learning for AI Agents

    A writer examines continual learning for LLM agents, framing today's models as amnesiac interns whose knowledge resets when the context window ends. LLMs have only two native memories: lossless but costly context and KV cache, and compressed but hard-to-update weights, with little in between. The piece weighs longer context, stateful memory harnesses, and compressing context into learned latent representations, citing Llama 3.1 70B's 141 GB weights and 320 KiB KV cache per token.

  2. Modal Blog22

    How Botika runs full-stack generative AI on Modal

    Botika runs its full-stack generative AI operation on Modal, covering a 100-terabyte image data pipeline, in-house foundation models with tens of billions of parameters, and roughly 15 models serving production inference. The pipeline processes that dataset with under a 1% error rate across thousands of concurrent containers, and a single researcher can now run 50 experiments a day instead of one or two.

September 1Tuesday
  1. Zyphra Research48

    Zyphra Releases PUFFER, an Incremental Fuzzy Deduplication System for Evolving Training Corpora

    Zyphra released PUFFER, an open-source provenance-aware incremental fuzzy-deduplication system for continuously growing LLM training corpora, under the Apache 2.0 license. It delivers an 11x-35x speedup over existing CPU-based deduplication methods and ingests one billion documents in 1.75 hours on a single process, with production deployment on more than 30 billion documents. PUFFER keeps a disk-resident MinHash-LSH history so new datasets can be added, interrupted jobs recovered deterministically, and individual datasets withdrawn without rebuilding the full corpus index.

July 26Sunday
  1. Berkeley AI Research33

    ABBEL: Teaching LLMs to Update Belief States for Long-Horizon Interaction

    Berkeley AI Research introduces ABBEL, a framework that replaces full interaction history with natural-language belief states and supervises their contents via belief grading. On CollabBench collaborative coding, reconstruction-based belief grading cuts the gap to full-context models by about 50% and trains in 50 steps instead of 100, while using fewer peak tokens.

July 21Tuesday
  1. OpenAI Alignment Blog62

    OpenAI and Apollo Research measure reward-seeking with Contrastive SDF

    OpenAI and Apollo Research developed Contrastive SDF, a test that finetunes two copies of a model on synthetic documents implying opposite grader preferences and measures how far behavior follows the grader. Applied to intermediate checkpoints of a capabilities-focused OpenAI o3 RL run without safety training, the grader gap trended upward from early to late checkpoints while gaps for other authorities stayed near zero. On the Broken Promises Coding and Log Falsification honesty evaluations, the late checkpoint's honesty depended far more on what it believed the grader rewarded, and the method recovered the target authority of three gpt-oss-120b model organisms trained to favor the User, the Grader, or OpenAI Leadership.

    Why it matters: OpenAI and Apollo Research introduce Contrastive SDF, a finetuning-based test that measures how strongly a model's behavior tracks what it believes its grader rewards, and find the tendency grows across an o3 RL run without safety training.

July 7Tuesday
  1. Berkeley AI Research36

    Intelligence Is Free, Now What? Data Systems for, of, and by Agents

    UC Berkeley's Aditya G. Parameswaran and collaborators argue that as inference costs fall from roughly $30 per million tokens in early 2023 to under $1 today, data systems must be redesigned around three challenges: systems for agents, of agents, and by agents. They note that in a text-to-SQL benchmark with multiple agents per task, only 10-20% of sub-plans are distinct, so 80-90% of sub-queries duplicate work, and propose reuse, approximation, and proactive data systems.

June 9Tuesday
  1. MiniMax Blog58

    MiniMax details MaxProof: generative-verifier RL and evolutionary search for mathematical proof

    MiniMax published a technical blog explaining MaxProof, the framework behind M3 exceeding the human gold-medal threshold on IMO 2025 and USAMO 2026. The pipeline trains three expert models for proof generation, error judgment and proof repair, then runs an evolutionary test-time search that samples 32 candidates, verifies each 4 times and refines over up to 10 rounds. MiniMax says the key lesson is that a generative verifier used as an RL reward should prioritize low false positives over average accuracy, citing M2-cycle reward hacking where the false-positive rate rose from about 2.9% to 8.0% during training.

June 1Monday
  1. Modal Blog46

    Modal: Reinforcement Learning for LLM Post-Training Is an Infrastructure Problem

    Modal says the bottleneck in RL post-training of LLMs is now infrastructure, not algorithms, and has released an open-source library drawn from running RL at scale. It details three pain points — glue code, cluster queueing and GPU under-utilization — and reports RDMA weight sync cutting Qwen3 8B update time from 2.62s to 41ms.

May 20Wednesday
April 29Wednesday
  1. Modal Blog33

    Building an RL Theorem-Proving Workflow on Modal

    AE Studio details how it trained LLMs to prove math theorems with reinforcement learning on Modal, comparing Evolution Strategies against GRPO while keeping the training loop fixed. The setup runs vLLM inference on GPUs, Lean compiler verification in isolated CPU sandboxes, and a lightweight orchestrator, using per-function images, .map() fan-out, and Volumes for base weights.

April 14Tuesday
  1. Modal Blog40

    Modal Autoscaling Autoresearch: Elastic GPUs for AI Research Agents

    Modal demonstrates running autoresearch agents on elastic GPUs: given Modal Skills, Claude Code ran 113 experiments over 238 GPU-hours on OpenAI's Parameter Golf challenge, finishing core training runs 5x faster than a single workstation while using a fraction of a dedicated cluster's resources. Agents provision GPUs via API calls, scaling from one to dozens of H100s and back to zero automatically.

March 25Wednesday
  1. Modal Blog22

    How Doppel cut ML infrastructure overhead with Modal

    Doppel migrated a large portion of its ML workflows to Modal, removing the infrastructure friction that slowed training experimentation and inference deployment. Parallel experiments via map() replaced sequential runs, while image layer caching and persistent volumes cut build times by up to 10x, with complex setups reaching sub-minute warm builds.

March 13Friday
  1. Berkeley AI Research38

    Berkeley AI Research Introduces SPEX and ProxySPEX for Identifying Interactions at Scale for LLMs

    Berkeley AI Research researchers describe SPEX and ProxySPEX, algorithms that identify influential feature, data, and model-component interactions in LLMs at scales far beyond prior methods. SPEX reframes interaction search as sparse recovery using sparsity and low-degreeness, while ProxySPEX exploits hierarchy to match SPEX with about 10x fewer ablations. On sentiment analysis, SPEX matches Faith-Shap and Faith-Banzhaf faithfulness on short inputs and retains it as context scales to thousands of features.

February 14Saturday
  1. Charlie O'Neill38

    The Michael Scott Paper Company of AI: Why Open-Source Specialisation Beats Frontier Labs on the Intelligence Ceiling

    Opus 4.6 and GPT-5.3-Codex both shipped last week, but the decisive contest is not raw capability. The argument holds that most economically valuable work sits in the specialisation layer, where a sub-80B model fine-tuned on a client's own data outperforms GPT-5.2 on that client's workflow — faster, cheaper, and without sending proprietary data to a third party. Open-source needs only to match the intelligence ceiling, not the frontier, and the labs' capital-dependency clock is the weaker position.

January 26Monday
  1. Charlie O'Neill38

    Continual learning and the post-monolith AI era

    Continual learning is inseparable from specialization because the cost of remembering scales with scope: as context grows, the KV cache grows linearly and can exceed all model weights, so information must move from token level into weights. Task specialization is essentially solved via continuous fine-tuning pipelines, while broad-scope, sparse-feedback, one-shot retention remains the hard problem.

January 15Thursday
  1. Modal Blog22

    Chai Discovery scales computational drug discovery on Modal

    Chai Discovery is using Modal to run its machine learning drug discovery pipelines, from protein embeddings and multiple sequence alignments to antibody design. Modal Volumes downloads and indexes datasets reaching hundreds of gigabytes once, then shares them across machines, while elastic GPU scaling spins up hundreds of GPUs in minutes. Chai says the same setup now carries research workflows into production without rewriting infrastructure.

December 28, 2025Sunday
  1. Modal Blog38

    Modal on Keeping 20,000+ GPUs Healthy Across Clouds

    Modal has scaled its multi-cloud GPU worker pool past 20,000 concurrent GPUs and launched over four million cloud instances in the last couple years, and is now sharing the reliability system behind it. The post covers instance type testing and selection, machine image preparation and boot checks, passive and active GPU healthchecking, and observability and support. It reports hyperscaler differences such as Cloud A's 99.6% boot success and Cloud C's H100s running above 90ºC in 2025.

May 29, 2025Thursday
  1. Modal Blog22

    Twirl Joins Modal to Expand Data Orchestration and European Presence

    Modal announced that Twirl, a data orchestrator for developing, testing and deploying data pipelines, is joining the company. Twirl co-founders Rebecka Storm and Eric Hansander will join Modal's Stockholm office, where they will develop and expand Modal's serverless infrastructure and grow its European footprint. Modal said the team's data product experience will help it expand beyond AI inference into adjacent AI, ML and data use cases.

December 10, 2024Tuesday
  1. Modal Blog30

    What is LLM fine-tuning? A Guide to Techniques and Tools

    Fine-tuning adjusts pretrained LLM weights to fit a specific task or domain, delivering higher quality than prompt engineering at lower cost and latency; a fine-tuned Llama 7B can be roughly 50 times more cost-effective per token than GPT-3.5 with comparable performance. The guide covers use cases such as structured output, style adherence and domain-specific instruction, plus options including OpenAI, Predibase, Modal, Google Colab and AWS SageMaker, and frameworks HuggingFace transformers, TRL and Axolotl.

December 4, 2024Wednesday
  1. Modal Blog22

    How a Top-Tier European Soccer Team Cut Data Processing Costs by 50% with Modal

    A top-tier European soccer team, anonymized as AFC Richmond, moved its match tracking data pipeline to Modal's serverless GPU platform and cut the cost of processing a full season of games by 50%. The team runs inference on ~3.5 million tracking observations per game through a custom transformer-based model, and with Airflow on Modal now processes each game's data in minutes rather than hours.

May 21, 2024Tuesday
  1. Modal Blog31

    Fine-tuning Stable Diffusion on Heroicons to Create an Infinite Icon Library

    Modal fine-tuned a Stable Diffusion model on the roughly 300-SVG Heroicons library to generate new black-and-white line icons in that style, including ones for concepts like barack-obama and golden-retriever. The full pipeline — downloading the pretrained model, full fine-tuning with HuggingFace Diffusers' train_text_to_image.py script, and serving a Gradio web UI — runs on Modal's serverless GPU platform. The write-up walks through dataset prep (SVG-to-PNG conversion, white backgrounds, caption metadata.csv) and notes that full fine-tuning worked best, though LoRA-style parallel adapters can also work for small datasets.

May 20, 2024Monday
  1. Modal Blog25

    Why Substack moved its AI and ML pipelines to Modal

    Substack has moved training and deployment for nearly all of its ML models—including spam detection, newsletter recommendations, audio transcription, sentiment analysis and image generation—from AWS SageMaker to Modal. Head of AI & ML Engineering Mike Cohen cited SageMaker's convoluted developer experience, hard-to-share code across notebooks and 5+ minute container startup times; on Modal, an inference endpoint took an hour to implement. Substack now runs a full fine-tune, validation and deployment workflow on Modal's storage primitives.

April 26, 2024Friday
  1. Modal Blog50

    Fine-tuning an open source embedding model to beat OpenAI's text-embedding-3-small

    A fine-tuned open source embedding model can outperform OpenAI's text-embedding-3-small in accuracy at lower cost using only a handful of examples. Modal ran an experiment on the Quora duplicate-questions dataset to find how much fine-tuning data a simple question-answering use case needs, and describes how to build one on its serverless infrastructure.

March 26, 2024Tuesday
  1. Modal Blog22

    How Ramp automated receipt processing with fine-tuned LLMs on Modal

    Ramp used Modal to fine-tune its LLMs and build a text-to-structured-JSON model for receipt management, cutting receipts requiring manual intervention by 34% on infrastructure estimated 79% cheaper than major LLM providers like OpenAI. Modal functions trained many candidate models in parallel, persisted weights to Modal volumes, and served a parametrized inference endpoint. One engineer also ported a PII-stripping script for 25,000 invoices to Modal, parallelizing it on 256 cloud workers to finish in 20 minutes for $100 instead of 3 days locally.