Sparse Goodness Functions Improve Forward-Forward Accuracy 30.7 Points on Fashion-MNIST
Sparsity in FF goodness functions identified as critical design choice, delivering 87.1% Fashion-MNIST accuracy via top-k and entmax methods.
New research shows sparse goodness functions substantially improve the Forward-Forward algorithm.
According to the primary source, top-k goodness which measures only the k most active neurons improves Fashion-MNIST accuracy by 22.6 percentage points over sum-of-squares (arXiv:2604.13081). Entmax-weighted energy adds learnable sparse weighting based on alpha-entmax, and separate label feature forwarding yields 87.1 percent accuracy, a 30.7 point gain.
The paper analyzed 11 different goodness functions, two architectures, and sparsity ranges, identifying sparsity as the dominant design factor with alpha approximately 1.5 best (arXiv:2604.13081). Original FF coverage focused on sum-of-squares without exploring this space (Hinton arXiv:2212.13345).
This connects to equilibrium propagation (Scellier, Bengio arXiv:1602.05179) and entmax sparsity techniques (Peters arXiv:1905.05737), pointing to broader trends in biologically plausible, hardware-efficient neural training.
AXIOM: Sparse top-k measurements in Forward-Forward learning reduce required activations for training, supplying an efficient biologically plausible substitute for backpropagation on specialized hardware.
Sources (3)
- [1]Sparse Goodness: How Selective Measurement Transforms Forward-Forward Learning(https://arxiv.org/abs/2604.13081)
- [2]The Forward-Forward Algorithm: Some Preliminary Investigations(https://arxiv.org/abs/2212.13345)
- [3]Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation(https://arxiv.org/abs/1602.05179)