Skip to content
Hot eventDeveloping

Lost OCaml generators for Apache DataSketches CPC tables reconstructed

1 reports1 reporting sourcesUpdated 50 min ago

Start here

AI digest

Tison Kun published a write-up reconstructing the lost OCaml generator code behind the hardcoded Huffman, permutation, polynomial and confidence tables in Apache DataSketches' CPC sketch. The original modeling code was written by Kevin J. Lang, who died in April 2019, on a machine that no longer exists; a Rust port of CpcSketch had shipped the tables without any generator source. Because the checked-in tables and their 20-digit header comments served as a strict numerical oracle, each candidate generator could be tested immediately: it either reproduced every codeword, permutation entry and floating-point diagnostic within IEEE-754 precision, or it was wrong. The reconstruction became PR-355, which adds the generators and derivations to datasketches-rust as unit tests. All 23 Huffman tables and 16 permutation tables now have bit-exact constructive tests, while the Monte Carlo confidence tables document that unit tests do not verify empirical cell values or true coverage. The million-trial simulator and raw outputs were kept locally rather than checked into the PR. The port has run in ScopeDB production for over six months, and serialized CPC cuts storage 30% to 40% versus HLL at equal accuracy.

Generated from reports · updated 44 min ago

Report timeline

Follow the reports to see the event from different sides.

November 16
  1. Hacker News · AI
    Reverse-Engineering Kevin J. Lang's Lost OCaml Code in Apache DataSketches

    A Rust port of Apache DataSketches' CpcSketch left its hardcoded Huffman and ICON lookup tables without any generator source, since Kevin J. Lang wrote the original modeling code in OCaml on a machine that no longer exists. Lang, who died in April 2019, authored the CPC algorithm and co-authored the KLL sketch. The port has run in ScopeDB production for over six months, and serialized CPC cuts storage 30% to 40% versus HLL at equal accuracy.

Heat for this event

Not enough continuous observations yet to draw a trend.