Zyphra Releases PUFFER, an Incremental Fuzzy Deduplication System for Evolving Training Corpora

Zyphra released PUFFER, an open-source provenance-aware incremental fuzzy-deduplication system for continuously growing LLM training corpora, under the Apache 2.0 license. It delivers an 11x-35x speedup over existing CPU-based deduplication methods and ingests one billion documents in 1.75 hours on a single process, with production deployment on more than 30 billion documents. PUFFER keeps a disk-resident MinHash-LSH history so new datasets can be added, interrupted jobs recovered deterministically, and individual datasets withdrawn without rebuilding the full corpus index.