Skip to content
Hacker News · AI·· 2 h agoAI score60

Contextflo test: four short notes lift Claude Haiku 5.5 agent accuracy on DABstep from 36% to 76%

Show HN: Company docs didn't help our AI agent. Four shortnotes doubled accuracy

AI summary

Contextflo ran a Claude Haiku 5.5 agent on the DABstep payments dataset from Adyen and Hugging Face and found company docs did not help accuracy, scoring 36% versus 40% with tables alone, while four short hand-written notes raised it to 76% on a held-out set of 42 questions at about 1.3 cents per try. The notes covered what blank fields mean, that a payment pays every matching price row, that monthly means calendar month, and how to answer cheapest-option questions. The authors also report that one overly broad note broke a question that previously passed, and that their initial 77% figure was revised to 63% on the exploration set after stricter scoring.

Source:Hacker News · AI · contextflo.com