Harmony Harness Posts Higher Secure-Code Scores on Held-Out Tests

Harmony Harness, a framework that changes the software surrounding AI coding agents, scored above the strongest listed alternatives on two held-out secure-code benchmarks, showing gains on tasks outside its development set.

Researchers affiliated with Google Cloud AI Research, Google DeepMind and Carnegie Mellon University developed the framework. It evolves an agent’s prompts, tools, workflow and checks while leaving the underlying model’s weights unchanged. Functionality and security remain separate optimization goals.

Harmony Harness also separates reusable mechanisms from adjustments tailored to a particular benchmark. It selects and combines those components when an agent runs, rather than applying every change in every setting.

The evaluation used SecRepoBench and BaxBench. SecRepoBench had 30 tasks in each of three groups: evolution, held-out in-distribution and held-out out-of-distribution. BaxBench had 28 tasks in each group.

On the held-out out-of-distribution tasks, Harmony Harness recorded a secure-pass@1 rate of 66.7% on SecRepoBench and 53.6% on BaxBench, averaged across three runs. The strongest competing scores listed for those groups were 50.0% and 42.9%, respectively, leaving margins of 16.7 and 10.7 percentage points.

Secure-pass@1 measures whether a single agent trajectory passes both the functionality and security checks. It does not describe general AI safety. The evaluation used Gemini 3.6 Flash for both the harness proposer and the task-solving agent, except where otherwise specified.

The work sits alongside earlier research on changing an agent’s surrounding software while keeping its model fixed, including a prior framework for evolving agent harnesses. The Harmony Harness experiments focused on secure code generation; they included no blockchain-specific tasks and do not establish how the approach performs in other agent domains.

The study was submitted Saturday, Oct. 10, at 4:06:59 p.m. ET (20:06:59 UTC).