Google and Hong Kong Researchers Find Fewer False Approvals in AI Agent Tests

Researchers affiliated with Google and The University of Hong Kong developed VeriCUA, a verifier that produced fewer false approvals across three AI-agent benchmarks by checking whether requested changes persisted in apps or files.

The system turns task instructions into checkable requirements, then uses interface or code-based checks to look for evidence in an app or file. Online checks are read-only. When the evidence is inconclusive, VeriCUA can abstain instead of issuing a verdict.

The paper was submitted Oct. 29 at 1:45 p.m. ET (17:45 UTC). Its evaluation covered 116 AndroidWorld episodes, 361 OSWorld episodes and 169 ScienceBoard episodes, using each benchmark’s official evaluator as the reference label.

On AndroidWorld, VeriCUA recorded 90.5% accuracy and a 7.7% false-positive rate. An auto-generated verifier scored 82.4% accuracy and had a 16.9% false-positive rate. A false positive occurs when a verifier approves a task that was not completed.

VeriCUA’s false-positive rate was 13.9% on OSWorld, compared with 25.9% for an auto-generated verifier and 27.9% for a vision-language model judge. On ScienceBoard, the rates were 5.4%, 21.7% and 24.8%, respectively.

No verdict was issued in 18% of AndroidWorld episodes, 22% of OSWorld episodes and 30% of ScienceBoard episodes. Among episodes where it issued a verdict, accuracy was 95.1% on AndroidWorld, 88.7% on OSWorld and 91.6% on ScienceBoard. The evaluation counted abstentions as failures in its overall binary results.

In the first online evaluation, VeriCUA used 39,200 tokens over 97 seconds per pass. The auto-generated verifier took 60 seconds and used 16,200 tokens. The paper frames verification as a way to evaluate agents, curate training data and assign rewards.

“A trajectory can appear successful even when edits remain unsaved or an operation fails without a visible warning,” Qiushi Sun and the other authors wrote. The findings come from benchmark environments; the paper does not describe a production deployment.