Skip to content
Only First-party#Data / training
November 4Wednesday
  1. Ai2 · Allen Institute for AI38

    Ai2 and Google collaborate on NeuralGCM+ climate models

    Ai2 has launched a climate modeling initiative backed by three-year funding from Google.org and technical support from Google Research to develop NeuralGCM+, an open-source model for faster, more reliable climate projections. The project extends Google Research's NeuralGCM, which combines physical equations for large-scale dynamics with AI for smaller-scale processes like cloud formation and rainfall, to simulate atmosphere and ocean together. Ai2 aims to cut regional temperature and rainfall errors over the past 50 years and release models and frameworks with documented workflows for outside researchers.

  2. Meta Engineering Blog34

    Meta F3 Lightning Accelerates Ads ML Feature Releases ~3.6x

    Meta introduced F3 Lightning, an infrastructure layer on top of its Facebook Feature Framework (F3) that decouples feature releases from shared execution, so each of the 20,000+ ads ranking features is compiled, validated, deployed, and rolled back independently. Comparable extraction features now reach production ~3.6x faster than prior F3 with no increase in infrastructure compute cost.

  3. Anthropic Research66

    Anthropic biology lab reports first prokaryotic vault particle structure, solved with Claude

    Anthropic's biology research lab reports the first 3D structure of a vault particle found in prokaryotic cells, a phage protein called L05 built from 96 copies with a mass of about 9.5 megadaltons. Under the team's direction, Claude searched a database of about 15 billion predicted proteins, designed DNA constructs, optimized a size-exclusion chromatography isolation that cut purification from three days to about 90 minutes, and ran RELION cryo-EM processing that produced a first full model in 34 hours. The preprint has been submitted to bioRxiv, and the team says the vaults' biological function remains unknown.

    Why it matters: Anthropic's biology lab reports the first 3D structure of a vault particle from prokaryotes, with Claude handling database search, construct design, purification and cryo-EM processing.

November 3Tuesday
  1. Ai2 · Allen Institute for AI55

    Ai2 releases StateBench to test how language models track state changes across long documents

    Ai2 introduced StateBench, a benchmark with four difficulty levels for long-context state tracking, from immutable assignment to self-referential and cross-referential updates. In tests, Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna handled the easier levels but dropped to 13% or lower on the longest general finite problems, while 275M-parameter Gated DeltaNet and hybrid models trained from scratch generalized almost perfectly on the first three levels and transformers never learned the hardest one.

  2. Apple Machine Learning Research32

    Attention to Mamba: A Recipe for Cross-Architecture Distillation

    Apple researchers propose a two-stage recipe for distilling a Transformer into a pure Mamba architecture without any Attention blocks. The method first distills a pretrained Transformer into a linearized Attention model via an adapted kernel trick, then distills that into an adapted Mamba model. The resulting distilled Mamba retains most of Pythia-1B's performance, reaching 14.01 perplexity versus the teacher's 13.86, with some gaps on tasks relying on in-context recall.

November 1Sunday
  1. Sakana AI Blog40

    Sakana AI Symposium: Jürgen Schmidhuber on AI's Past and Future, Japan's Push into Language Models, World Models and Physical AI

    Sakana AI held its "From World Models to Real-World AI" symposium in Tokyo on October 26, 2026, where Chief Scientific Advisor Jürgen Schmidhuber gave a keynote and Sakana AI announced it will develop its own pretrained language model series, Sakana Sawara (tentative name). The first small model, Sawara Small (tentative name), is planned for release in spring 2027 if development proceeds smoothly. More than 400 attendees joined the event, which also featured a conversation between Schmidhuber and CEO David Ha.

October 30Friday
  1. Microsoft Research40

    Microsoft builds synthetic enterprise data platform to evaluate Copilot

    Microsoft's Industry Evaluation Data Generation Platform creates full-scale synthetic enterprise environments to evaluate Microsoft Copilot without using customer data. Grounded in public sources such as O*NET occupational data, job postings, and SEC filings, it generates realistic tenants with users, meetings, emails, and documents, plus automated query pipelines and quality frameworks. Microsoft says this establishes the first standardized evaluation baseline for Copilot.

  2. Microsoft Research22

    Microsoft Research Focus Edition #97: ForkPilot, CABRA, TeleTune and AI Use at Work

    Microsoft Research Focus Edition #97 highlights ForkPilot, a self-improving agentic framework that matches the best existing methods while cutting AI processing demands by up to 59% across six benchmarks. The edition also covers CABRA, a framework showing coding agents still struggle to understand unfamiliar codebases, and TeleTune, which learns reusable skills from user activity logs. It also examines Copilot Chat activity, where document editing exceeded 20% of classified activities.

  3. Amazon Science38

    Speeding up LLMs for travel applications: Medusa plus knowledge distillation

    Combining knowledge distillation with the Medusa framework lets a 1.1B-parameter TinyLlama student match the quality of its 70B-parameter Llama-3.1 teacher while running 10 to 20 times as fast at less than one-third the cost. Working with Booking.com on AI Trip Planner, Smart Filters and accommodation description generation, the pipeline closed over 90% of the quality gap and added 2 to 3.6x further speedup from Medusa heads.

October 29Thursday
  1. Meta Engineering Blog40

    Meta Andromeda: Meta's Next-Gen Ads Retrieval Engine Hits 1B Ads per Second

    Meta detailed the next generation of Andromeda, its personalized ads retrieval engine, which scores over one billion ads per second in production — a 100x throughput lift over the 2024 system. The new architecture brings LLM-inspired mixture-of-experts to retrieval, approaching ranking-level model performance at roughly 300x fewer inference-time FLOPs, with over 10 billion ads per second on a single NVIDIA GH200 GPU. Since 2025, Andromeda has delivered a 2.4% increase in conversions on Facebook.

  2. Preferred Networks News40

    Zenrin Group and Preferred Networks Form Business and Capital Alliance to Fuse Geospatial Data with AI

    Zenrin, Zenrin DataCom and Preferred Networks have signed a business and capital alliance, with the Zenrin companies subscribing to a PFN third-party allotment. The partners will combine continuously updated geospatial data, including historical change records, with PFN's generative AI, physical AI and 3D spatial recognition technology to build AI models that understand and predict real-world change. Initial focus areas include public sector, real estate, retail and distribution, targeting demand forecasting, store site selection and urban planning, disaster prevention and logistics simulation.

October 28Wednesday
  1. Ai2 · Allen Institute for AI40

    How researchers are adapting olmOCR 2 for historical and archaeological archives

    Researchers are adapting Ai2's open olmOCR 2 model to transcribe historical and archaeological archives. Historian Maurice Casey built ArtificeOCR, which runs olmOCR 2 locally with Tropy to make his archival photo library keyword-searchable, while TU Wien's CENTURIA project fine-tuned olmOCR 2 with LoRA on 57 annotated records, cutting transcription error on 450 test records from 15.58% to 1.11%.

  2. Google Research58

    Google Research's sleep framework lets LLMs consolidate new knowledge without catastrophic forgetting

    Google Research introduced a sleep framework for continual learning that alternates active interaction with offline memory consolidation, letting LLMs absorb new facts into expanded Mixture-of-Experts parameters instead of losing prior knowledge. The approach combines parameter expansion, on-policy distillation with reinforcement learning-based imitation, and a dreaming stage that generates and filters synthetic training samples. The authors report near-zero performance drop in continual translation for unseen languages such as Manchu and Kalamang, near-perfect scaling to 10M tokens on BABILong, 48.9% single-task and 46.2% CPT results on SQuAD-based factual incorporation, and 79.2%, 69.0% and 46.1% on AIME 24/25 and HMMT 25.

  3. Meta AI Research Blog30

    Meta Supports AI Research at Five US Universities to Accelerate Medical and Scientific Breakthroughs

    Meta is supporting research at five US universities — Caltech, Ohio State University College of Medicine, Louisiana State University, Indiana University, and UTHealth Houston — with access to its latest Muse models, researcher collaboration, and funding. Researchers will use Muse Spark Open, Muse Wonder, Muse Spark, the 30B open-weight Muse Glimmer, plus SAM, V-JEPA, DINO, UMA, and SeamlessM4T. Projects span medicine, physics, chemistry, biology, computer science, and mathematics, with universities retaining control of all patient and clinical data.

October 27Tuesday
  1. Ai2 · Allen Institute for AI58

    Ai2 study tests whether LLMs can reliably judge research idea novelty

    Ai2 released Old Ideas, Novel Problems, a study and benchmark testing how reliably LLMs evaluate the novelty of research ideas, along with data, code and a pipeline for building new evaluation sets. Across six judges including GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6, small prompt changes shifted accuracy by more than 50 percentage points on AI-generated ideas, and a simple prompted Claude Opus 4.6 baseline scored 0.854 macro-F1 at $2.59 versus 0.642 at $30.66 for Idea Novelty Checker and 0.772 at $82.05 for Sakana AI's AI Scientist.

  2. Anthropic Research65

    Anthropic reports early progress on using Claude to accelerate drug development

    Anthropic's life sciences team has spent several months working with six drug developers, including Bristol Myers Squibb, Generate Biomedicines and Divergence Labs, plus running its own discovery programs, to test where Claude can speed up drug development. Reported early results include a Generate design roughly 70% shorter than the standard version that let immune cells make about 10 times as many receptors, and roughly 35,000 antibody designs generated largely autonomously for Divergence in about a month. Anthropic says the work is still mostly preclinical and plans to extend Claude into clinical trials as the programs advance.

    Why it matters: Anthropic's first detailed account of embedding Claude in real drug programs shows where the model helps and where it still falls short across six partners.

October 23Friday
  1. Ai2 · Allen Institute for AI31

    LitPivot: Ai2's System Helps Researchers Pivot Research Ideas Through Literature

    Ai2 and University of Pennsylvania researchers built LitPivot, a system that organizes retrieved papers around the facet of a research proposal being edited and suggests literature-backed revisions. In a study with 17 researchers, domain experts rated ideas developed with LitPivot as better grounded in the literature than those built with a text editor and an AI chat tool. Participants also reported their revised ideas as more novel and useful, and LitPivot will be presented at UIST 2026.

  2. Amazon Science38

    The science behind TorchNeuron, Amazon's new PyTorch backend for AWS Trainium chips

    Amazon released TorchNeuron, an open-source PyTorch backend that translates PyTorch code into operations executable on AWS Trainium systems with minimal or no code changes. It adds an opt-in adaptive eager mode that groups and compiles operations for near-compiled performance, plus multi-level NEFF caching; a shared compilation service cut time to first training step by about a third in a 64-process LLM test. The Annapurna team tuned and benchmarked it using 841 representative models distilled from hundreds of thousands of open-source models.

October 22Thursday
  1. Modal Blog71

    Modal releases Spindle, an open-source Tinker-compatible training server

    Modal released Spindle, an open-source, Tinker-compatible training server that runs in a customer's Modal workspace, letting teams customize the stack and pay for GPU time instead of tokens. Spindle supports full-parameter training and multi-LoRA across open models, and Modal reports that sharing one trainer across 8 clients cut cost per 1M output tokens by up to 80% versus Tinker on a DAPO-math GRPO job. Modal says Spindle is still in beta and positions it between its Dojo training library and raw cluster primitives.

    Why it matters: Modal's open-source Spindle server offers teams a Tinker-compatible alternative that trades per-token pricing for GPU time, with multi-LoRA sharing reported to cut costs by up to 80%.

  2. Ai2 · Allen Institute for AI58

    Ai2 releases OlmoEarth v1.3 satellite embedding model with global precomputed embeddings and a browser explorer

    Ai2 released OlmoEarth v1.3, a satellite embedding model that turns a year of Sentinel-1, Sentinel-2 and Landsat time series into one 128-dimensional vector per 10-meter pixel, with the first 64 dimensions usable as a smaller embedding. Ai2 precomputed annual embeddings for the whole planet and published them on Source Cooperative, and its 128-dimensional embeddings scored highest on 20 of 27 evaluations against AlphaEarth Foundations and Tessera v2, including a new tropical crop benchmark called PLANTEUR. The release also includes the OlmoEarth Embeddings Explorer, a no-sign-in browser tool for comparing years, finding similar places and training small classifiers on the spot, with 2025, 2024 and 2023 global and 2017 to 2022 covering select countries; weights, pretraining data and code are open.

  3. inclusionAI Models58

    inclusionAI releases Ling-3.0-flash-Sante, a 124B healthcare-specialized MoE model

    inclusionAI released Ling-3.0-flash-Sante, a healthcare-specialized model built on Ling-3.0-flash with 124B total parameters and 5.1B active parameters per token, with weights on Hugging Face and ModelScope. It uses a hybrid-linear MoE architecture combining Kimi Delta Attention, gated MLA and sparse MoE layers, trains to 256K tokens, and enables thinking mode by default with tool calling through compatible serving frameworks. The company labels it a research model, not a medical device, and says outputs must not replace professional medical advice, diagnosis or treatment.

  4. Microsoft Research64

    Microsoft Research and co-author prove the 31-year-old KLS conjecture

    Microsoft researcher Xinzhi Zhang and a co-author have proved the Kannan-Lovász-Simonovits (KLS) conjecture, a 31-year-old open problem in high-dimensional geometry, showing the bound holds with a universal constant independent of dimension. The proof grew out of a human-directed agentic process in which AI agents explored more than 100 candidate approaches while researchers pruned and redirected the search, and a separate unguided attempt with leading models on a comparable token budget fell short. Two other proofs appeared the same day, and a Lean 4 formalization was subsequently released.

    Why it matters: A 31-year-old open problem in high-dimensional geometry is resolved through a human-directed agentic search over roughly 100 candidate approaches, with independent proofs and Lean formalization as confirmation.

  5. Anthropic Newsroom28

    Anthropic Opens Call for Guest Investigators in the Life Sciences

    Anthropic is inviting principal investigators and their research groups at academic institutions, nonprofits, and government labs to apply for 8–12 week guest investigator projects using Claude in areas such as causal human biology, structural and computational molecular biology, medicinal chemistry, drug discovery, and clinical science. Anthropic will run its models and harnesses at no cost, and outputs will be published with open-source data and code while investigators retain rights to their data. Applications are open for approximately 14 days, with selected participants notified by email about 14 days after the close.

October 21Wednesday
  1. Modal Blog58

    Modal releases Dojo, a library for production-grade LLM training

    Modal released Modal Dojo, a training library that provides building blocks, tuned default recipes and an observability dashboard for production-grade LLM post-training. It builds on open-source frameworks Miles and Slime, supports models from half a billion to multi-trillion parameters, and exposes an API and CLI that agents can drive to launch and debug runs. Modal says Axiom Bio used it to scale to 64k-context DeepSeek V4.1 Flash runs on 64 B300s, and Afterquery trains Kimi and DeepSeek at 256k context length.

  2. Apple Machine Learning Research26

    TEAL: Token-Space Efficient Active Learning for Low-Budget Semantic Segmentation

    Apple researchers (with University of Waterloo) introduce TEAL, the first pixel-level active learning framework built on frozen ViT representations, selecting diverse candidates directly on the native token lattice via a two-level MaxHerding strategy plus margin-based uncertainty refinement. With a single labeled pixel per image, TEAL beats the strongest baseline by up to +24.36 mIoU on CamVid, Cityscapes, ADE20K and Pascal Context, reaching up to 97% of fully supervised performance with just 10 px.

October 20Tuesday
  1. Google Developers Blog60

    Google 将 TorchTitan 接入 TorchTPU，并发布 TPU 模型优化手册

    Google released a TPU fork of Meta's TorchTitan that runs PyTorch training on Cloud TPUs through TorchTPU's native torch.device("tpu") backend, with TPU-native kernels and a step-by-step optimization playbook. On TPU v7x (Ironwood), Llama 3 405B holds about 57% MFU from 128 to 1,024 chips, Qwen3 235B reaches 32.42% MFU on one 1,024-chip slice, and FLUX.1 reaches 19.4% MFU on 1,024 chips at 93% weak-scaling efficiency. The fork is at https://github.com/google-pytorch/torchtitan/, and a Transformers modeling backend plus an agent onboarding skill extend the same trainer to other checkpoints.

    Why it matters: Google 与 Meta 合作把 TorchTitan 移植到 TPU，并给出从 FLUX.1 到 Qwen3 235B 的逐级调优路径与 MFU 数字，可作为 PyTorch 团队评估 TPU 训练可行性的参考。

October 17Saturday
  1. Google Research71

    Google Research outlines VERID and closed-loop agent workflows for autonomous wet labs

    Google Research describes three projects aimed at grounding AI agents in real wet-lab experimentation, including the VERID framework, a semantic digital twin of a lab workcell built on the open-source PyLabRobot framework that tracks deck layout, well volumes and tip inventory and dry-runs protocols before execution.

    Why it matters: Google Research outlines three projects that connect AI agents to real wet-lab hardware, showing how simulated dry runs and closed-loop experiments let agents correct their own design rules.

October 15Thursday
  1. Mistral AI71

    Mistral releases Timestral time-series foundation models, topping GIFT-Eval, TIME and fev-bench

    Mistral released Timestral, its first family of time-series foundation models, in four sizes from 5M to 209M parameters. The largest model ranks first on all three main public benchmarks, GIFT-Eval, TIME and fev-bench, among pre-trained zero-shot models from a single checkpoint without ensembling, using two-thirds of the parameters of the previous state of the art. Timestral produces probabilistic multivariate forecasts with 21 quantiles, supports past-only and known-future covariates, runs on CPU without a GPU, and is available through Mistral Forge for fine-tuning, with a GitHub repository and technical report published under a modified MIT licence.

    Why it matters: Mistral's first time-series foundation model family claims top zero-shot rankings on GIFT-Eval, TIME and fev-bench with a 209M-parameter checkpoint, offering a size-accuracy tradeoff reference for forecasting work.

  2. Goodfire Research62

    Goodfire research essay argues neural networks are already clean and superposition was misread

    Goodfire researcher argues that the canonical evidence for polysemanticity and superposition in neural networks was misread, and that what looked like arbitrary packing is mostly a motif he calls recursive generalization. Retraining InceptionV1 with ordinary architectural improvements and no change to model size or training data made 99-100% of neurons in its last layer easily interpretable, and preliminary audits of Qwen3-32B estimate roughly 90% of neuron poles have one main job. The essay concludes networks use one learning principle rather than two, so interpretability can proceed by studying neurons and circuits directly instead of first training decomposition models such as sparse autoencoders.

    Why it matters: A first-hand reinterpretation of the field's canonical polysemanticity evidence, arguing that networks generalize recursively rather than pack unrelated concepts, which would make direct mechanistic study viable.

October 13Tuesday
  1. Modal Blog58

    Modal tests mixed-precision, mixed-accelerator RL samplers on Qwen3.6-35B-A3B

    Modal trained Qwen3.6-35B-A3B on agentic coding with samplers spread across six accelerator types (B200, B300, H100, H200, A100, RTX PRO 6000) at NVFP4, FP8 and BF16 precision, feeding one BF16 trainer on B200. Plain GRPO collapsed on both mixed and homogeneous samplers, but adding masked importance sampling plus score centering, with or without top-p mask replay, trained stably to the end and reached SWE-bench Pro V2 pass@4 on par with homogeneous samplers, at roughly 15% lower sampler cost per update. The mixed samplers ran with an average sampler-trainer KL 4.7x that of the homogeneous setup, and the authors argue stability depends on keeping the mismatch bounded rather than low.

  2. Amazon Science75

    Amazon Web Services researchers propose priming to convert pretrained Transformers into hybrid state space models without full retraining

    Amazon Web Services researchers introduce priming, a method that reuses pretrained Transformer weights to cut the pretraining cost of hybrid state space architectures by more than 99%, using under 0.5% of the source model's pretraining token budget. In controlled comparisons, Gated KalmaNet outperformed Gated DeltaNet and Mamba-2, and a 32B GKA-based hybrid matched its source Transformer on reasoning benchmarks while decoding up to twice as quickly at 128,000 tokens of context. The team released 8B and 32B primed hybrid models, training code, Triton kernels, and a vLLM serving plugin under the Apache 2.0 License.

    Why it matters: Amazon Web Services researchers show that reusing pretrained Transformer weights can cut hybrid architecture pretraining cost by over 99%, making controlled comparisons of state space layers practical.

October 8Thursday
  1. Zyphra Research36

    Zyphra Research: Expert Coupling Cuts MoE Pretraining Communication

    Zyphra Research shows MoE routing is highly structured, with co-selected experts correlated within a layer and a token's expert choices predicting later-layer routing. Exploiting this via correlated expert placement and token shuffling cuts all-to-all communication time up to 2.63× and improves end-to-end training step time up to 1.41× in Megatron-LM experiments on MI300X GPUs, without changing architecture or routing decisions.

  2. Waymo Blog30

    Waymo Study: Sober Drivers Still Face Nearly 4x Nighttime Fatal Crash Risk

    Waymo's peer-reviewed research finds that even fully sober drivers face 3.1 to 3.9 times higher nighttime fatal crash risk than in daytime, and that removing alcohol-impaired drivers cuts average urban fatal crash involvement by 23%, from 1.42 to 1.10 per 100 million miles. A second study of 37,654 fatal cases in the 2023 FARS database shows only 8.2% involved strictly human behavior factors alone. Waymo frames the findings as support for a Safe System Approach and for benchmarking autonomous driving against a zero-BAC human baseline.

October 2Friday
  1. Meta AI Research Blog71

    Meta shares six math papers co-authored with mathematicians and Muse Spark

    Meta published six mathematics papers produced in collaboration with mathematicians who used Muse Spark 1.1 and 1.2 in Thinking Mode through the regular meta.ai chat interface, with no custom research scaffold. Five of the papers answer previously open questions, covering Gaussian ellipsoid fitting thresholds, finite-time blow-up for a biharmonic nonlinear Schrödinger equation, a 384-element counterexample to a group-theory conjecture, tightness of a cycle-based relaxation in binary polynomial optimization, and a link between string two-point functions and height functions on curves. Each paper marks which passages were primarily drafted by researchers and which by AI, credits prior work, and acknowledges independent concurrent solutions from other teams.

    Why it matters: Meta 公布六篇由数学家与 Muse Spark 协作完成的论文，其中五篇给出此前开放问题的答案，并标注 AI 与人类各自撰写的段落。

October 1Thursday
  1. Zyphra Research38

    How Hybrid NoPE Models Learn Position Without Position Encodings

    Zyphra Research explains how hybrid language models recover positional information even when global attention layers use no explicit position encoding. Local layers such as sliding window attention and gated linear attention create a recency bias in the residual stream, and global NoPE attention learns query and key projections that read it out as an implicit relative position encoding. The findings offer a mechanistic account of why interleaving local mixing with global NoPE attention works well and point to positional representations that extrapolate beyond training sequence lengths.

  2. Goodfire Research58

    Goodfire develops protein-embedding biosecurity monitors for AI agents

    Goodfire Research developed sequence-aware biosecurity monitors that use protein language model embeddings to flag concerning DNA or protein sequences in agent workflows. On a custom dual-use benchmark built around protein toxins, the monitors outperformed frontier model safeguards with fewer refusals on benign requests, reached roughly 0.98 AUROC on paraphrases whose parent proteins were in the screening database, and ran in milliseconds per sequence. The authors report that detection improves as protein models get more capable, and note the paraphrase results rely on in-silico estimates and do not establish whether redesigned proteins retain biological activity.

September 4Friday
  1. Modal Blog22

    Markov Robotics Uses Modal to Run End-to-End Robot Training

    Markov Robotics is training a world model for zero-shot general-purpose robot dexterity, using fewer than ten hours of robot data where comparable results typically need thousands of hours. The company runs nearly all its compute on Modal, from preprocessing hundreds of terabytes of video to 2–10 concurrent training runs and remote inference via Modal Tunnels at roughly 20 milliseconds round-trip latency. Preprocessing that could take a full day completes in twenty to thirty minutes, with fan-out to upwards of 500 GPUs at once.

September 2Wednesday
  1. Modal Blog22

    How Botika runs full-stack generative AI on Modal

    Botika runs its full-stack generative AI operation on Modal, covering a 100-terabyte image data pipeline, in-house foundation models with tens of billions of parameters, and roughly 15 models serving production inference. The pipeline processes that dataset with under a 1% error rate across thousands of concurrent containers, and a single researcher can now run 50 experiments a day instead of one or two.

September 1Tuesday
  1. Zyphra Research48

    Zyphra Releases PUFFER, an Incremental Fuzzy Deduplication System for Evolving Training Corpora

    Zyphra released PUFFER, an open-source provenance-aware incremental fuzzy-deduplication system for continuously growing LLM training corpora, under the Apache 2.0 license. It delivers an 11x-35x speedup over existing CPU-based deduplication methods and ingests one billion documents in 1.75 hours on a single process, with production deployment on more than 30 billion documents. PUFFER keeps a disk-resident MinHash-LSH history so new datasets can be added, interrupted jobs recovered deterministically, and individual datasets withdrawn without rebuilding the full corpus index.