Skip to content
Hacker News · AI·· 4 h agoAI score58

Show HN: Benchmark diffs the whole database after a computer-use agent's task

Show HN: A benchmark that diffs the whole database after an AI agent's task

AI summary

A new open-source benchmark, software-world-model, grades computer-use agents by diffing the entire database after each task in self-hosted Invoice Ninja, Dolibarr, Odoo 17 and Odoo 18, yielding four verdicts: success, collateral damage, irreversible change and false completion. In 635 scored Phase 0 episodes, task success held from Odoo 17 to Odoo 18 (78% to 76%) while collateral damage rose from 15% to 21% and irreversible changes from 6% to 10%, and clean success ran up to 12 points below raw success. A 3-step composite task succeeded 0 of 50 times across 4 apps and 2 agents, and the author notes small samples, one run per task and wide Wilson 95% intervals.

Source:Hacker News · AI · github.com