ArXiv 2610.20667: Pairwise judging of saturated transcripts cuts capability prediction error 5.3x
ArXiv 2610.20667 demonstrates that transcript-level protocols extract capability signal from saturated benchmarks, with pairwise judging delivering the largest measured gain. The signal is section-specific and distribution-shift sensitive. Methods must now target robustness across reasoning-model transitions.
The study simulates saturation on historical releases and tests three transcript protocols against unsaturated items. Pairwise judging outperforms single-transcript scoring, deterministic features, and embeddings. Signal concentrates in reasoning and tool-call sections yet remains distributed across the full transcript. Performance drops when protocols encounter the first reasoning models released in early 2025, indicating distribution-shift sensitivity.
Saturated benchmarks retain measurable capability information beyond final accuracy. The 5.3x error reduction holds only within similar model families; cross-family generalization fails on novel reasoning traces. Existing extraction methods therefore extend benchmark lifetime but require explicit robustness testing against architectural shifts before use in safety cases.
Operational implication is immediate: frontier labs can mine existing saturated runs for ranking signals without new evaluations. Regulators and auditors gain a documented lower bound on transcript utility while recognizing the documented 2025 generalization gap. Next required step is protocol variants that explicitly model reasoning-trace novelty rather than relying on current judging templates.
Jehyeok Yeon: Pairwise transcript judging will exceed 0.85 Spearman correlation on 2027 reasoning-model releases by December 2027.
Sources (3)
- [1]Primary Source(https://arxiv.org/abs/2610.20667)
- [2]Supporting Source(https://arxiv.org/abs/2303.08774)
- [3]Supporting Source(https://arxiv.org/abs/2412.14170)