Skip to content
Hot eventDeveloping

Open-source benchmark diffs full database to grade computer-use agents

1 reports1 reporting sourcesUpdated 4 h ago

Start here

AI digest

Shubham Joshi released software-world-model, an open-source benchmark that grades computer-use agents by diffing the entire database after each task rather than trusting the agent's own report. It runs in self-hosted Invoice Ninja, Dolibarr, Odoo 17 and Odoo 18 and returns four verdicts: success, collateral damage, irreversible change and false completion. In 635 scored Phase 0 episodes, task success held roughly flat from Odoo 17 to Odoo 18 (78% to 76%), while collateral damage rose from 15% to 21% and irreversible changes from 6% to 10%. Clean success ran up to 12 points below raw success. A three-step composite task succeeded 0 of 50 times across four apps and two agents. Joshi cautions that these are small-sample results from one person's Phase 0: 36 to 72 episodes per cell, one run per task, with wide Wilson 95% intervals. Setup requires Docker, Python 3.10–3.12 (or uv) and about 8 GB of free disk; everything binds to 127.0.0.1, agents cannot leave localhost, and no real email is sent. Joshi reports GPT-6 Luna cost one to three cents per successful task in Phase 0.

Generated from reports · updated 3 h ago

Report timeline

Follow the reports to see the event from different sides.

November 16
  1. Hacker News · AI
    Show HN: Benchmark diffs the whole database after a computer-use agent's task

    A new open-source benchmark, software-world-model, grades computer-use agents by diffing the entire database after each task in self-hosted Invoice Ninja, Dolibarr, Odoo 17 and Odoo 18, yielding four verdicts: success, collateral damage, irreversible change and false completion. In 635 scored Phase 0 episodes, task success held from Odoo 17 to Odoo 18 (78% to 76%) while collateral damage rose from 15% to 21% and irreversible changes from 6% to 10%, and clean success ran up to 12 points below raw success. A 3-step composite task succeeded 0 of 50 times across 4 apps and 2 agents, and the author notes small samples, one run per task and wide Wilson 95% intervals.

Heat for this event

Current, comparable range
9
Peak, comparable range
10November 16 13:00
Change in 24 hours
–

The trend compares the same participants over hours observed continuously, so its range can be narrower than the current heat count. Hover or tap the chart for hourly heat; the arrow keys move through it.