TURTLE Raises Share of Hard AppWorld Tasks by 165% in Tests

Researchers at Stanford University and Google DeepMind developed TURTLE to evolve AI tool-use benchmark tasks, raising the share of hard AppWorld tasks by 165% relative to the original set’s 19.8%.

Nearly all 735 evolved AppWorld tasks had solver success rates between 40% and 60%. In the original AppWorld set, 69% of tasks were solved consistently.

TURTLE uses three AI agents to revise and test tasks. A proposer suggests a change, a solver attempts the revised task, and a feedback generator uses the result to guide another round. Changes can add filters, extra steps or elements that are less visible within a task. For a task to pass the solvability check, it had to reach a 50% completion rate across at least 10 attempts.

The team tested TURTLE on AppWorld, τ-Bench and AutomationBench. The τ-Bench work included τ-telecom and τ-banking. The environments cover software tasks involving email, payments and collaboration. The main reported results used Gemma 4 31B-it as the solver.

Tool-use benchmarks measure whether AI agents can operate software to complete tasks that take multiple steps. As models improve, an unchanged test set can become less useful for measuring their performance. Creating and checking multistep tasks by hand also takes researchers’ time. TURTLE is designed to revise benchmark tasks automatically and target an average completion rate near 50%, where tasks remain challenging while still being solvable.

The main results were tuned using Gemma 4 31B-it. How well the evolved tasks maintain the same difficulty for other models remains an open question.