Format-Aware Fusion delivers 37.9K tokens/s/GPU on Llama-3 8B FP4 pretraining

The arXiv:2610.00053 paper introduces format-aware fusion that aligns quantization producers, scale domains, and consumer layouts for MXFP row-wise, global NVFP, and CTA-local NVFP. Llama-3-family 8B models were pretrained to 160 billion tokens with bfloat16 output projections and compiled cross-entropy loss. Matched-accelerator runs isolate the fusion effect from hardware variables.

Throughput data show the fastest custom MXFP route at 37.9K tokens/s/GPU and a row-gradient stochastic rounding variant with fixed-sign Hadamard preconditioning at 37.2K, achieving 86.3 percent bfloat16 FLOP utilization. The same configuration finished 2.11 percent above the raw bfloat16 loss endpoint. A Transformer Engine baseline ending with four bfloat16 blocks reached only 27.1K tokens/s/GPU and 0.87 percent above bfloat16 loss. Downstream task rankings diverged from training-loss order, indicating joint dependence on scale contract, operand packing, and execution path.

Operationally, the method reduces the overhead of scale computation and layout construction that previously nullified FP4 Tensor Core gains. It extends patterns seen in earlier MXFP and integer quantization work by enforcing producer-consumer co-design at the kernel level. This removes the need for final bfloat16 blocks in some layers while preserving convergence, lowering memory traffic and enabling longer context or larger batch sizes on the same silicon.

Next steps include porting the fusion kernels to Blackwell FP4 paths and measuring end-to-end energy per token at 405B scale.