Contextflo: four short notes lift Claude Haiku 5.5 agent accuracy to 76%
Start here
Contextflo ran a Claude Haiku 5.5 agent on DABstep, a public payments dataset from Adyen and Hugging Face, and reported that the company's own documentation did not help: accuracy was 36% with docs versus 40% with tables alone. Four short hand-written notes raised accuracy to 76% on a held-out set of 42 questions, at about 1.3 cents per attempt. The notes covered what blank fields mean, that a payment applies to every matching price row, that "monthly" means calendar month, and how to answer cheapest-option questions. Contextflo also reported that one overly broad note broke a question that had previously passed, and that its initial 77% figure was revised to 63% on the exploration set after stricter scoring. The authors say they did not test a larger database and are not claiming how the result scales.
Generated from reports · updated 1 h ago
Report timeline
Follow the reports to see the event from different sides.
- Hacker News · AIContextflo test: four short notes lift Claude Haiku 5.5 agent accuracy on DABstep from 36% to 76%
Contextflo ran a Claude Haiku 5.5 agent on the DABstep payments dataset from Adyen and Hugging Face and found company docs did not help accuracy, scoring 36% versus 40% with tables alone, while four short hand-written notes raised it to 76% on a held-out set of 42 questions at about 1.3 cents per try. The notes covered what blank fields mean, that a payment pays every matching price row, that monthly means calendar month, and how to answer cheapest-option questions. The authors also report that one overly broad note broke a question that previously passed, and that their initial 77% figure was revised to 63% on the exploration set after stricter scoring.
Heat for this event
Not enough continuous observations yet to draw a trend.