C++ CUDA repo ships paged KV cache and online-softmax attention for Llama 3.2 1B

The repository contains a minimal C++17 and CUDA engine that parses model.safetensors metadata, maps BF16 weights to GPU, and executes RMSNorm, RoPE, grouped-query attention, and SwiGLU MLP kernels for both prefill and decode. A shared page pool with per-sequence block tables supports static and continuous batching while admission logic reserves pages for prompt plus output tokens up to the 2048-token context limit. All components are covered by unit tests and 80+ pages of handwritten diagrams.

The implementation follows the paged-attention design introduced in the 2023 vLLM paper (arXiv:2309.08872) yet restricts scope to a single fixed 1B model and single GPU, omitting the multi-request throughput measurements and prefix caching present in the original vLLM codebase. Direct comparison with the Llama 3.2 technical report (arXiv:2407.21783) shows the engine correctly reproduces the 2048-wide hidden state, 16-head grouped-query attention, and tied embedding layout.

Operationally the engine demonstrates that online softmax accumulation across non-contiguous pages eliminates the need for contiguous KV gathers during decode, reducing memory fragmentation for variable-length requests. Persistent cuBLAS handles and reused activation buffers keep peak memory close to the KV pool size plus model weights.

Next milestones listed in the repository include multi-GPU tensor parallelism and extension of the KV pool beyond 128 MiB while preserving the same page-table interface.