Skip to content
Hot eventDeveloping

Audit finds AgentDojo mislabeled 433 attack runs as safe

1 reports1 reporting sourcesUpdated 2 h ago

Start here

AI digest

An audit of AgentDojo, a benchmark for testing whether AI agents resist prompt-injection attacks, found that 433 runs were scored as "no attack" even though the agents carried out the attacker's action. The audit also says it missed 55 of 744 cases. The audit's author has released a tool that re-runs AgentDojo findings on a chosen model and seals the evidence for offline checksum verification; it is available on PyPI. The findings come in a preprint that has not been peer reviewed, so the error rates are the auditor's own claims rather than independently confirmed results.

Generated from reports · updated 2 h ago

Report timeline

Follow the reports to see the event from different sides.

November 20
  1. Hacker News · AI
    Audit finds AgentDojo scored 433 runs as no attack despite agents executing the attacker's action

    2 it missed 55 of 744. x replies it scored as attacked as warnings. 0 on PyPI) that audits verdicts, re-runs AgentDojo findings on a chosen model, and seals evidence for offline checksum verification; the preprint is not peer reviewed.

Heat for this event

Current, comparable range
9
Peak, comparable range
10November 20 01:00
Change in 24 hours
–

The trend compares the same participants over hours observed continuously, so its range can be narrower than the current heat count. Hover or tap the chart for hourly heat; the arrow keys move through it.