Open-source benchmark diffs full database to grade computer-use agents
Start here
Shubham Joshi released software-world-model, an open-source benchmark that grades computer-use agents by diffing the entire database after each task rather than trusting the agent's own report. It runs in self-hosted Invoice Ninja, Dolibarr, Odoo 17 and Odoo 18 and returns four verdicts: success, collateral damage, irreversible change and false completion. In 635 scored Phase 0 episodes, task success held roughly flat from Odoo 17 to Odoo 18 (78% to 76%), while collateral damage rose from 15% to 21% and irreversible changes from 6% to 10%. Clean success ran up to 12 points below raw success. A three-step composite task succeeded 0 of 50 times across four apps and two agents. Joshi cautions that these are small-sample results from one person's Phase 0: 36 to 72 episodes per cell, one run per task, with wide Wilson 95% intervals. Setup requires Docker, Python 3.10–3.12 (or uv) and about 8 GB of free disk; everything binds to 127.0.0.1, agents cannot leave localhost, and no real email is sent. Joshi reports GPT-6 Luna cost one to three cents per successful task in Phase 0.
Generated from reports · updated 3 h ago
Report timeline
Follow the reports to see the event from different sides.
- Hacker News · AIShow HN: Benchmark diffs the whole database after a computer-use agent's task
A new open-source benchmark, software-world-model, grades computer-use agents by diffing the entire database after each task in self-hosted Invoice Ninja, Dolibarr, Odoo 17 and Odoo 18, yielding four verdicts: success, collateral damage, irreversible change and false completion. In 635 scored Phase 0 episodes, task success held from Odoo 17 to Odoo 18 (78% to 76%) while collateral damage rose from 15% to 21% and irreversible changes from 6% to 10%, and clean success ran up to 12 points below raw success. A 3-step composite task succeeded 0 of 50 times across 4 apps and 2 agents, and the author notes small samples, one run per task and wide Wilson 95% intervals.
Heat for this event
- Current, comparable range
- 9
- Peak, comparable range
- 10November 16 13:00
- Change in 24 hours
- –
The trend compares the same participants over hours observed continuously, so its range can be narrower than the current heat count. Hover or tap the chart for hourly heat; the arrow keys move through it.