Amazon-Affiliated Study Finds AI Orchestrator Raises Success at Higher Cost

Researchers affiliated with Amazon and the University of Washington found a redesigned AI-agent coordinator improved enterprise-workflow task success, but it used nearly five times the tokens and took three times as long.

The research team submitted its preprint Nov. 7, arguing that a single tool-using agent can serve as a practical starting point for enterprise workflows. The findings point to a choice shaped by each standard operating procedure (SOP), or multistep work process, rather than one architecture that leads in every setting.

The controlled comparison tested four architecture families across eight configurations, including a function-calling agent, a skill-equipped agent, centralized orchestration and a triage-style agent group. Each configuration ran 2,124 tasks using Claude Opus 4.8, with the model, tasks and tools held fixed.

The single-agent baseline completed 67.4% of tasks, using 24,900 tokens and averaging 24 seconds per task. One centralized orchestration setup had a 58.2% success rate. A redesigned centralized orchestrator reached 75.6% success, with 117,300 tokens and an average task time of 73.5 seconds.

The redesigned system's success rate was 8.2 percentage points above the baseline, while its token use was about 4.7 times as high and its average task time about 3.1 times as long. Its success rate was the highest among the tested configurations.

Skill results also varied: a high-quality skill setup reached 73.1% success, compared with 59.8% for a degraded-skill setup and 67.4% for the baseline. A procedure-by-procedure comparison found no statistically significant difference between the high-quality and degraded-skill setups.

The experiment used deterministic mock tools, so it did not measure delays, errors or unpredictable behavior from live services. The experiment tested one model; its results may not carry over to production workflows or other model families.

Reported sample totals and success rates conflict. The reported figures conflict: one set lists 13 SOPs, with success rates of 67.4% for the baseline and 75.6% for redesigned orchestration; another lists 12 SOPs and rates of 65.6% and 74.4%. Only the redesigned setup significantly beat the baseline, but the statistical figures conflict: one set gives an odds ratio of 1.72 and p = 0.007, while another gives 1.78 and p = 0.008.