XYBench: A Benchmark for AI Assistants Facing Misguided User Questions

Hugging Face released XYBench, an 8K+ question benchmark testing whether AI assistants catch the misconceptions behind users' requests, and it won the Best Resource Paper Award at EMNLP 2026. Built with Carnegie Mellon, NVIDIA, and Johns Hopkins from 8,115 real-world questions, it found Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro all leaned toward the stated request over the user's actual goal.