MARCH multi-agent judge declares solutions equal on 78-95% of code pairs
Multi-agent code judging collapses to near-random equivalence calls because candidate solutions are not independent evidence. Label-free measurements extracted from pipeline logs identify ungrounded decisions and allow selective refusal that raises accuracy while answering half the items. The findings directly constrain reliability of LLM judges in automated code review and verification workflows.
The arXiv paper tests the published MARCH pipeline on code judging tasks. Evidence independence fails because the two candidate solutions serve as the sole verification source for each other. Log-derived measurements show the pipeline cannot distinguish candidates on the majority of items, independent of problem difficulty or judge size.
Label-free gating on one internal measurement lets the system decline 50 percent of comparisons and lifts accuracy from 20.7 to 36.9 percent. Direct queries retain higher accuracy on the same items. The result isolates when evidence is insufficient rather than improving the underlying judge.
Task-oriented deployments that rely on LLM judges for code review or pull-request triage now face measurable refusal rates. Production pipelines must add explicit decline logic or accept systematic equivalence declarations that carry no grounding signal.
Follow-on work will need to quantify how often real code corpora provide independent evidence before multi-agent verification can be treated as reliable.
MARCH: Accuracy after gating will stay below 40 percent on new code benchmarks released before Q3 2027.
Sources (2)
- [1]Primary Source(https://arxiv.org/abs/2609.30328)
- [2]Supporting Source(https://arxiv.org/abs/2307.09288)