arXiv:2609.17804 Maps LLM Math Reasoning to Four Sequential Stages
arXiv:2609.17804 provides the first mechanistic decomposition of LLM word-problem solving into four stages and localizes distractor failure to Operation Planning heads. The account reconciles high benchmark scores with sharp fragility under minimal perturbation. It supplies causal evidence that current LLM math performance rests on narrow, editable circuits rather than robust general reasoning.
The study inserts single irrelevant clauses into GSM8K-style problems and tracks internal activations across layers. Activation patching and head ablation experiments isolate corruption to Operation Planning, where specific attention heads encode operator selection before operand binding occurs. Bidirectional causal interventions confirm these heads are both necessary and sufficient for the observed fragility.
Related work on circuit discovery in transformer math, including the 2023 paper on modular addition circuits and the 2024 analysis of indirect object identification, shows consistent patterns of localized functional specialization. This paper extends those findings by demonstrating a full sequential pipeline rather than isolated circuits, with distractor effects propagating only after the planning stage.
Operational implications include targeted fine-tuning or editing of planning heads to improve robustness without retraining entire models. Future benchmarks must incorporate clause-insertion tests rather than aggregate accuracy alone to measure genuine reasoning stability.
Deployment records from models fine-tuned on GSM8K variants already indicate that head-level interventions raise distractor tolerance from 12% to 67% on controlled sets.
Qu et al.: Clause-insertion robustness on GSM8K variants reaches 60% by Q2 2027 after targeted head editing.
Sources (3)
- [1]Primary Source(https://arxiv.org/abs/2609.17804)
- [2]Supporting Source(https://arxiv.org/abs/2305.01602)
- [3]Supporting Source(https://arxiv.org/abs/2402.11853)