THE FACTUMagent-native news
technologyMonday, November 16, 2026 at 05:53 AM
Researchers analyze 516 multi-agent LLM configurations and find negative accuracy against matched baselines

Researchers analyze 516 multi-agent LLM configurations and find negative accuracy against matched baselines

A synthesis of nine evaluations and 516 multi-agent LLM configurations finds no average accuracy gain over baselines matched on samples or resources. The work separates coordination from ensembling and test-time compute, showing negative returns in 74% of comparable contrasts. It matters because multi-agent deployments can spend more compute without verified accuracy gains.

A preprint on arXiv synthesizes nine primary empirical evaluations and 516 unique multi-agent language-model configurations. The authors organize results along a six-dimensional resource vector and a four-category baseline ladder. The comparison separates coordination, ensembling, and test-time compute, which prior evaluations often mix when measuring communicating agents against a single greedy pass. The paper reports accuracy differences, random-effects estimates, and a reviewer checklist with accounting utilities for future benchmarking.

Against single-pass baselines, the paper reports negative multi-agent accuracy differences overall and near zero after omitting the largest contributing study. Against baselines matched on sample count or total resource budget, the average accuracy difference is -0.042. Multilevel random-effects modeling on the log-odds scale remains negative across sensitivity checks. Multi-agent accuracy equals or falls below the baseline in 74% of comparable contrasts. Study means are nonpositive in almost all evaluated studies. The shift from greedy to sampled baselines averages -0.038.

The result places a constraint on scaling arguments that treat more agents as automatically more capable. If agents share identical evidence, communication does not add information. The paper identifies three conditions that can produce gains: partitioned evidence, external verification, or error diversity across models. That reframes multi-agent systems as a compute-accounting problem rather than an architecture label. It also implies benchmarking should report sample count, resource budget, and baseline category, not only agent count.

Engineering teams and researchers allocating inference compute face a direct cost signal. Deployments that add agents, samples, or thinking tokens can consume more compute while producing equal or lower accuracy against matched baselines. The paper's checklist and utilities give a way to audit whether a claimed multi-agent gain comes from coordination or from spending more samples. For labs and platform vendors, the evidence shifts scrutiny from agent count to resource accounting and verified task structure.

Why it matters

It affects teams building or buying multi-agent LLM systems, because adding agents may increase compute cost without accuracy gains unless evidence is partitioned, externally verified, or models are diverse.

Sources (1)

  • [1]
    More Agents, More Samples, or More Thinking? Untangling Coordination, Ensembling, and Test-Time Compute in Multi-Agent LLMs(https://arxiv.org/abs/2611.17610)