# Google DeepMind’s XYEval Finds Relative AI Performance Falls Up to 46.7%

By Simon Yoon

Canonical URL: https://www.tokenpost.com/news/technology/40859
Published: 2026-10-26T07:52:23.000Z
Updated: 2026-10-26T07:52:23.000Z
Section: Technology

> The benchmark tests five models across six task suites. Its largest relative performance drop was for Gemini 3.1 Pro on Terminal-Bench.

Google DeepMind researchers introduced XYEval, a benchmark for testing whether AI agents follow misleading user suggestions, after finding that the advice cut relative task performance by as much as 46.7%.

The largest decline was recorded for Gemini 3.1 Pro on Terminal-Bench. XYEval examines the “XY problem,” in which a user focuses on a proposed solution, or X, instead of the underlying goal, or Y. The test asks whether an agent can recognize the goal and still solve the task when the instruction points it toward a misleading approach.

The benchmark keeps each task’s environment and ground-truth answer fixed while modifying its instruction. The misleading advice is framed as a confident suggestion, and the task remains solvable. In SWE-bench, the suggestion can replace an existing user direction rather than being added to it.

The researchers tested five models across six benchmarks: τ2-bench, SWE-bench Verified, SWE-bench Pro, Terminal-Bench, Humanity’s Last Exam and MCP-Atlas. For τ2-bench, rules written by human experts generated the misleading suggestions.

On τ2-bench, performance fell further in a simulated “pedantic” user condition. In that setup, the user insisted on receiving an explanation before accepting a better solution. A basic system instruction telling agents to watch for XY problems reduced the effect only partly.