Prompt Checklist Cuts False Code Reversals in Gemini Tests, but On-Demand Betting Trails

Google-affiliated researcher Nghi D. Q. Bui’s prompt checklist cut false code reversals in Gemini benchmarks, while its On-Demand Betting mode had lower pooled pass and repair rates in mixed-feedback coding tasks.

Bui submitted the preprint Oct. 25 at 8:35 p.m. ET (Oct. 26 at 00:35 UTC). It compares five Gemini models on eight public benchmarks spanning coding, report and calculation review, academic reasoning, financial analysis and command-line work. The method, called Thinking-in-Bets, has an agent record checkable claims, supporting evidence and its confidence before reading feedback. The agent then uses that record to decide whether to keep its work, examine a specific objection or revise a claim it considers uncertain.

Across SWE-Together and SWE-Interact, the rate of correct code requirements undone after mistaken criticism fell from 12.2% (744 of 6,074 requirements) with the standard agent to 0.02% (one of 6,130) with the checklist method. The measure captures one specific failure: changing correct work in response to an incorrect objection.

In a 100-trial review of reports and calculations, 89 deliverables passed after checklist-based review, compared with 58 for the standard agent. The standard agent corrected 24 of 25 flawed drafts; the checklist method corrected 23 of 25.

Results were weaker for On-Demand Betting in pooled mid-session coding tests across four Flash models. The standard agent passed 35 of 80 tasks and corrected 25 of 45 valid bug reports. On-Demand Betting passed 29 of 80 tasks and corrected 18 of 39 valid reports. It used 85% fewer checklist tokens than the Always-On version.

Results also varied by model. In a two-run test of Gemini 3.8 Flash, On-Demand Betting corrected 17 of 23 valid bug reports, compared with 12 of 23 for the standard agent. It passed 21 of 40 tasks, while the standard agent passed 23 of 40.

Scores on the closed-book Vals Finance benchmark rose from 29.0% to 34.2% for Gemini 3.8 Flash and from 44.6% to 52.5% for Gemini 4.0 Argon. The benchmark assessed financial analysis, not crypto trading or live financial decisions.

The interactive tests used simulated users who typically made brief objections without a failing example. The work tested only Gemini models and used hand-picked thresholds; 25 of 85 headline comparisons remained significant after a multiple-testing adjustment. It describes a prompt-level experiment, not a Gemini product launch or a change to Google’s public service.