Bahdanau 2014 Attention Operation Removes Fixed-Context Bottleneck in Neural Translation

Bahdanau, Cho, and Bengio replaced the single fixed-length vector with per-decoder-step alignment weights computed over encoder hidden states. The resulting model trained on WMT'14 English-French data produced equivalent performance on long and short sentences, eliminating the length-dependent drop documented in Sutskever et al. 2014. The operation required no architectural overhaul beyond addition of a context vector at each step.

The 2017 Vaswani et al. transformer replaced recurrent layers entirely with scaled dot-product attention and multi-head projections, cutting training FLOPs by more than an order of magnitude on the same WMT task while raising BLEU. Subsequent scaling studies (Kaplan et al. 2020) showed loss continuing to fall with compute under the same attention formulation, confirming the mechanism's compatibility with larger models rather than any inherent intelligence property.

Operationally, attention supplies dynamic routing without recurrence, allowing parallel training and constant-time access to any prior token. This property directly enabled the 175-billion-parameter GPT-3 checkpoint and all later decoder-only models. No subsequent architecture has displaced attention on language-modeling benchmarks through 2025.

Next deployments will test whether attention remains sufficient once context windows exceed 1 million tokens; current evidence from LongRoPE and Ring Attention shows quadratic cost still dominates unless sparsity is added.