Specialist AI Agent Network Scores 70% on GAIA Benchmark

A coordinated network of specialized AI agents scored 70.13% on the GAIA general assistant benchmark, beating the single-developer systems tested and showing an advantage for coordinated specialists on that mix of tasks.

The strongest specialist system working alone scored 35%. The network finished 12 percentage points above the median general-purpose single-developer system and three points above the best one. Some comparison systems used multiple agents internally.

The Oct. 14 preprint involved researchers at Princeton University and UC Berkeley. It examined whether systems developed separately could perform better when their work was coordinated, rather than testing one AI model against a group of models.

For the GAIA evaluation, an AI development process generated and refined agents representing 54 specialist developers and nine generalist developers. The compared groups had the same total development budget, although each general-purpose system had several times the budget of an individual specialist.

The specialist network broke a request into smaller jobs, selected agents for those jobs and combined their answers. GAIA tests general assistant tasks that can require reasoning, web browsing, tools and work with files, images or audio. Its broad range of tasks gave the network opportunities to draw on different specialists.

A separate evaluation produced a different result. On SealQA, a benchmark focused on web search, the coordinated network performed worse than a routed single-developer system and incurred additional coordination costs. That evaluation used 42 specialist developers and 11 generalist developers.

The routed comparison handled a full request through a single-developer system and could try another system before returning an answer. A selected system could itself use one agent or several. The SealQA result therefore concerns how the systems were developed and coordinated, rather than a simple count of agents answering a question.

The developers in both evaluations were simulated through the AI development process. The experiment did not involve independent companies offering agents through a live service, so its benchmark results do not establish how such providers would work together in practice.

The GAIA gain and SealQA shortfall show that the benefit of coordination varied with the task being tested. A live network of independent providers would also need to address payment sharing, security, privacy and accountability.