arXiv:2608.21362 reports 4.49x TTFT reduction via chunk-level KV reuse on Qwen2.5-3B

KVBoost implements chunk-level KV cache reuse for decoder-only models through a dual-hash scheme that decouples positional prefix hashes from content hashes. It adds two boundary repair passes, SelectiveRecompute and CacheBlendRecompute, plus asymmetric int8/int4 quantization and importance-weighted eviction. The system targets RoPE-based Hugging Face checkpoints without architecture changes and operates under fixed memory budgets.

On the evaluated workload the method delivered a 4.49x TTFT reduction and a 16% improvement over standard prefix caching. Accuracy remained statistically unchanged. Chunk reuse worked for shared content at arbitrary offsets, a scenario where contiguous-prefix systems record zero hits. Adaptive boundary splitting further reduced recompute volume on long contexts.

Prior KV-cache work such as PagedAttention and vLLM focused on paging within a single sequence or exact prefix matches. KVBoost extends reuse to non-contiguous shared chunks and quantifies deviation-driven repair cost. This pattern appears in production traces where bug reports, documentation snippets, and test cases repeat across unrelated prompts.

Operational deployment requires only an inference-layer wrapper. Memory-bounded eviction and quantization keep the working set inside existing GPU budgets, enabling immediate rollout on current hardware without model retraining or custom kernels.