XAI-Arena Records Spearman's Rho 0.693 Against Human Ratings on Eight Explanation Dimensions
XAI-Arena supplies an LLM judge protocol that reproduces human quality rankings of XAI explanations at rho=.693. It removes subjective bottlenecks while preserving multidimensional stakeholder views. The method enables scalable, auditable comparisons across explanation techniques.
The framework pits multiple XAI methods against one another on fixed datasets and model families. LLMs receive stakeholder personas and output scalar scores plus rationales. Human validation on the same items yields the reported correlation, confirming that automated rankings preserve relative ordering without requiring fresh annotator pools for each study.
Existing XAI papers rely on small-N human surveys that vary by prompt wording and rater pool, blocking direct replication. XAI-Arena fixes the judge model, prompt template, and dimension definitions, enabling exact reproduction across labs. The benchmark already covers tabular, image, and text tasks, exposing consistent gaps: post-hoc feature attributions score lower on actionability than inherently interpretable models under the same judge.
Operationally, regulators and procurement teams can insert XAI-Arena into audit pipelines to generate reproducible quality deltas between candidate explanation methods. The 0.693 correlation threshold indicates the system is ready for comparative triage but still requires human review for high-stakes single decisions.
Next steps include expanding the judge ensemble and testing temporal drift when base LLMs are updated.
Feuerriegel: At least three peer-reviewed XAI papers will cite XAI-Arena as primary evaluation method within 18 months.
Sources (2)
- [1]Primary Source(https://arxiv.org/abs/2609.09428)
- [2]Supporting Source(https://arxiv.org/abs/2306.05685)