Spectral Entropy Collapse Precedes Grokking Transition
Normalised spectral entropy collapses prior to generalization in grokking, forming a predictive order parameter necessary but not sufficient for the transition (Truong et al. 2026; Power et al. 2022; Nanda 2023).
Normalised spectral entropy of representation covariance crosses a stable threshold before generalization in grokking (Truong et al., arXiv:2604.13123).
Grokking follows a two-phase pattern of weight norm expansion then entropy collapse, with normalised spectral entropy \tilde{H}(t) crossing \tilde{H}^* \approx 0.61 an average 1,020 steps prior to generalization in 100% of 1-layer Transformer runs on group tasks (Truong et al., arXiv:2604.13123). The relation holds for both \mathbb{Z}/97\mathbb{Z} and S_5, and a power-law \Delta T = C_1(\tilde{H}-\tilde{H}^*)^\gamma + C_2 predicts onset with 4.1% error (Truong et al., 2026; Power et al., arXiv:2201.02177).
Original grokking reports documented delayed generalization after memorization but did not isolate a scalar predictive order parameter or the two-phase dynamic (Power et al., arXiv:2201.02177). Coverage missed that entropy collapse, not weight norm, drives the shift: causal interventions blocking collapse delayed grokking by +5,020 steps while norm-matched controls did not (Truong et al., arXiv:2604.13123).
Mechanistic interpretability work established circuit-formation metrics during grokking (Nanda et al., progress measures analysis, 2023). Spectral entropy collapse occurs in MLPs yet does not produce grokking, establishing necessity without sufficiency and confirming architecture dependence (Truong et al., arXiv:2604.13123; Power et al., arXiv:2201.02177).
AXIOM: Spectral entropy collapse is a measurable leading indicator of grokking that precedes generalization by hundreds to thousands of steps and is necessary but not sufficient, depending on network architecture.
Sources (3)
- [1]Spectral Entropy Collapse as an Empirical Signature of Delayed Generalisation in Grokking(https://arxiv.org/abs/2604.13123)
- [2]Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets(https://arxiv.org/abs/2201.02177)
- [3]Progress Measures for Grokking via Mechanistic Interpretability(https://www.neelnanda.io/mechanistic-interpretability/grokking)