XYBench: Helping with the problem behind the question

XYBench is a new benchmark for evaluating how AI assistants handle misguided questions where the user's stated request is based on a misconception about their true goal. Using over 8,000 real-world questions, it measures whether models can identify the user's underlying goal and redirect them toward a more effective solution. Evaluations of models like GPT-5 and Claude Sonnet 4.6 show they often focus on the user's mistaken approach, failing to steer them toward the correct solution as human experts do. This research highlights a critical gap between an LLM's ability to recognize good advice and its ability to generate it, underscoring the challenge of inferring and acting on true user intent.