Ai2’s XYBench asks whether a model can get from a user’s requested fix to the problem that prompted it. Thicker frosting is a plausible answer to slipping icing; the cupcakes may simply be too warm. Across the study’s Stack Overflow examples, five models focused on the underlying goal in roughly 35–41% of free responses. In a separate multiple-choice test, six models recognized a goal-focused answer 71–84% of the time. An evaluation that hands the model its possible diagnoses measures a different skill from the one the user needs.
Why the cupcakes mattered more than the frosting
Models recognized a better diagnosis more often than they supplied it unprompted.
1 source · 1 min read
Link to noteOriginally published 16 November 2026. Reporting, forecasts, prices, odds and event availability reflect the original edition.
Sources & further reading
- Ai2’s study huggingface.co