Intermediate LLMLingua-2 rates cut Jetson AGX Thor RAG GPU energy 53.2%
Paper 2608.19535 quantifies adaptive context compression for edge RAG on Jetson AGX Thor. Intermediate LLMLingua-2 rates deliver up to 53.2% GPU energy reduction with negligible quality loss. Runtime telemetry policies are required to exploit the observed operating region.
The arXiv paper 2608.19535 measures full RAG pipelines on edge SoC hardware rather than isolated generation. Generation accounts for 90% of per-query latency and 91% of GPU energy at 7B-8B scale. Compressor overhead on the same die is quantified, showing that fixed-rate policies either leave savings on the table or degrade retrieval utility.
Data reveal a narrow adaptive window: mild pruning misses energy reduction while aggressive pruning increases end-to-end latency once quality drops force re-retrieval. Telemetry inputs such as KV-cache occupancy, current draw, and query token length determine the operating point at runtime.
Related work on LLMLingua-2 (arXiv 2403.12968) and KV-cache eviction policies demonstrates that static budgets ignore device state variation across Jetson power modes. The present measurements close that gap by coupling compression directly to live SoC sensors.
Next steps require closed-loop controllers that adjust compression ratio per query using the same telemetry streams already exposed by NVIDIA JetPack, enabling production deployment without external orchestration.
NVIDIA JetPack: runtime adaptive compression API ships in 36.5 if measured energy reduction exceeds 45% on production 7B workloads within 90 days of internal validation.
Sources (3)
- [1]Primary Source(https://arxiv.org/abs/2608.19535)
- [2]Supporting Source(https://arxiv.org/abs/2403.12968)
- [3]Supporting Source(https://docs.nvidia.com/jetson/archives/r36.3/DeveloperGuide/)