The Substack article by ADL Rocha questions the relentless pursuit of more compute and RAM in AI, proposing that innovations in mathematics and algorithms may hold the key to future breakthroughs. This aligns with observations from the field where diminishing returns on scaling have been noted (https://adlrocha.substack.com/p/adlrocha-what-if-ai-doesnt-need-more).

Primary sources like Kaplan et al.'s scaling laws paper initially fueled the compute-heavy approach, but subsequent work such as Hoffmann et al.'s Chinchilla study revealed that data and model size must be scaled in tandem, a nuance sometimes lost in popular coverage that overemphasizes hardware (https://arxiv.org/abs/2001.08361; https://arxiv.org/abs/2203.15556).

Furthermore, developments in efficient computing like the FlashAttention algorithm demonstrate concrete gains from mathematical insights, suggesting the community may have underinvested in such efficiency research in favor of brute-force scaling (https://arxiv.org/abs/2205.14135).