Ai2 and Hugging Face release XYBench to test AI handling of misconception-based requests

Ai2 and Hugging Face researchers released XYBench, a benchmark of 8,115 real-world questions that tests whether AI assistants recognize when a user's request rests on a misconception and steer them toward their actual goal, rather than answering the request as stated. The work, built with Carnegie Mellon, NVIDIA, and Johns Hopkins, won the Best Resource Paper Award at EMNLP 2026. In the released results, Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro all tended to follow the user's stated request rather than address the user's actual goal. XYBench currently covers English-language queries and mainly procedural problems; extending it to other languages, cultures, and conversational settings remains an open direction.

Generated from reports · updated 10 min ago

1. November 14 01:02 · 1 reports
   Release of XYBench benchmark for misconception-based questio
   Hugging Face Blog：XYBench: A Benchmark for AI Assistants Facing Misguided User Questions
2. November 13 16:00 · 1 reports
   Ai2 releases XYBench benchmark for misconception-based quest
   Ai2 · Allen Institute for AI：Ai2 releases XYBench, an 8K-question benchmark for the XY problem in AI assistant answers

Not enough continuous observations yet to draw a trend.