Audit finds AgentDojo mislabeled 433 attack runs as safe
Start here
An audit of AgentDojo, a benchmark for testing whether AI agents resist prompt-injection attacks, found that 433 runs were scored as "no attack" even though the agents carried out the attacker's action. The audit also says it missed 55 of 744 cases. The audit's author has released a tool that re-runs AgentDojo findings on a chosen model and seals the evidence for offline checksum verification; it is available on PyPI. The findings come in a preprint that has not been peer reviewed, so the error rates are the auditor's own claims rather than independently confirmed results.
Generated from reports · updated 2 h ago
Report timeline
Follow the reports to see the event from different sides.
- Hacker News · AIAudit finds AgentDojo scored 433 runs as no attack despite agents executing the attacker's action
2 it missed 55 of 744. x replies it scored as attacked as warnings. 0 on PyPI) that audits verdicts, re-runs AgentDojo findings on a chosen model, and seals evidence for offline checksum verification; the preprint is not peer reviewed.
Heat for this event
- Current, comparable range
- 9
- Peak, comparable range
- 10November 20 01:00
- Change in 24 hours
- –
The trend compares the same participants over hours observed continuously, so its range can be narrower than the current heat count. Hover or tap the chart for hourly heat; the arrow keys move through it.