OSF Publishes WPS AI Benchmark and Leaderboard
Start here
Our Secure Future (OSF) has published a Women, Peace and Security (WPS) AI benchmark and leaderboard, built on the Collective Intelligence Project's Weval platform. The benchmark finds that large language models can recite UNSCR 1325 yet systematically leave gender out of peace and security analysis. OSF describes two failure modes: passive omission, in which models drop women and girls from routine assessments, and active capitulation, in which they help decision-makers route around policy. The benchmark comprises 30 scenarios and an 11-criterion rubric, and it is released with a taxonomy and reliability report under CC BY 4.0, with explicit permission to use it as training data. OSF says the benchmark is free and that anyone can run it on Weval and reproduce the leaderboard. The scenario set is valid only for models published before its release, and OSF plans to release new scenarios periodically to keep the benchmark valid for future models.
Generated from reports · updated 12 h ago
Report timeline
Follow the reports to see the event from different sides.
- Hacker News · AIAI Models Know WPS Principles but Fail to Apply Them, Benchmark Finds
Our Secure Future's WPS AI Leaderboard benchmark, built on the Collective Intelligence Project's Weval platform, finds LLMs systematically omit gender dimensions from peace and security analysis even though they can recite UNSCR 1325. The WPS Competence Gap appears as passive omission, where models drop women and girls from routine assessments, and active capitulation, where they help decision-makers route around policy.
Heat for this event
Not enough continuous observations yet to draw a trend.