Ai2’s XYBench asks whether a model can get from a user’s requested fix to the problem that prompted it. Thicker frosting is a plausible answer to slipping icing; the cupcakes may simply be too warm. Across the study’s Stack Overflow examples, five models focused on the underlying goal in roughly 35–41% of free responses. In a separate multiple-choice test, six models recognized a goal-focused answer 71–84% of the time. An evaluation that hands the model its possible diagnoses measures a different skill from the one the user needs.

Ai2’s study