Config Variations Produce 58-Point Swings and Flip Model Rankings on ARC, MMLU, HellaSwag, TruthfulQA
The paper demonstrates that modern multiple-choice LLM benchmarks are not neutral measurement instruments. Harness configuration alone moves individual model scores across ranges wide enough to reorder entire leaderboards, and the items that drive those reorderings are precisely the high-discrimination items retained by compression techniques. The work supplies both the diagnostic grid and the data release needed to surface this dependence before ranks are published.
The fragility grid fixes weights, items, and greedy decoding while varying only prompt wording, option order, and scoring method (likelihood versus generation). On the 3,679 items from ARC, HellaSwag, MMLU, and TruthfulQA, four of the twelve models occupy first place under at least one equally defensible configuration. Item discrimination, the statistic maximized by benchmark-compression algorithms, correlates with fragility at 0.28, indicating that current pruning methods preferentially retain the very items whose correctness flips with harness details.
This pattern extends earlier findings in the 2021 MMLU paper and the HELM framework, both of which documented prompt sensitivity yet reported only aggregate variance. By resolving variance to individual items and matched model pairs, the grid shows that stable items produce ties; separation between adjacent models is manufactured almost entirely by harness-dependent responses. The dominant axis is not option permutation, which most protocols already standardize, but the choice between per-option likelihood and generated-text extraction.
Operationally, any leaderboard that publishes a single rank without a fragility audit therefore encodes an arbitrary harness decision rather than a stable capability ordering. The released per-item matrix allows any group to recompute the entire analysis on a CPU in seconds, supplying a concrete pre-publication check that existing evaluation suites lack.
Deployment pipelines that select models by these ranks therefore inherit the same instability when prompts or decoding methods differ from the reported harness.
HELM or LMSYS: within 9 months at least one major public leaderboard will publish fragility-band ranks alongside point estimates or be overtaken by a competing board that does.
Sources (3)
- [1]Primary Source(https://arxiv.org/abs/2608.21382)
- [2]Measuring Massive Multitask Language Understanding(https://arxiv.org/abs/2009.03300)
- [3]Holistic Evaluation of Language Models(https://arxiv.org/abs/2211.09110)