Counterfactual Routing Awakens Dormant Experts in MoE Models
Researchers introduce training-free Counterfactual Routing to activate dormant long-tail experts in MoE LLMs, improving factual accuracy 3.1% on benchmarks without added compute (arXiv:2604.14246).
Static Top-k routing in sparse Mixture-of-Experts models favors high-frequency patterns, leaving specialist experts for long-tail knowledge under-activated despite causal importance on other inputs (Hu et al., arXiv:2604.14246). Counterfactual Routing integrates layer-wise perturbation analysis with the Counterfactual Expert Impact metric to shift resources from syntax-dominant to knowledge-intensive layers while holding total activation count constant (Hu et al., arXiv:2604.14246).
Switch Transformers established MoE scalability via similar routing but left long-tail hallucination unaddressed (Fedus et al., arXiv:2101.03961); a hallucination survey documented factual errors on rare facts as systemic across LLMs (Huang et al., arXiv:2311.05232). The source understates how CoR's virtual ablation retrieves causally decisive experts without retraining, a connection to mechanistic interpretability techniques omitted in original coverage.
CoR yields 3.1% average factual accuracy gain on TruthfulQA, FACTOR, and TriviaQA with unchanged inference budget, outperforming static scaling on the Pareto frontier (Hu et al., arXiv:2604.14246).
AXIOM: Counterfactual routing directly targets under-activated experts in production MoE systems, a reliability gap that has limited deployment on knowledge-intensive tasks.
Sources (3)
- [1]Awakening Dormant Experts: Counterfactual Routing to Mitigate MoE Hallucinations(https://arxiv.org/abs/2604.14246)
- [2]Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity(https://arxiv.org/abs/2101.03961)
- [3]A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions(https://arxiv.org/abs/2311.05232)