PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference

2024/05/21 by Dongjie Yang, Yang, Dongjie, XiaoDong Han +10 · 1 voice · 55 citations
Computer Science · #Advanced Data Storage Technologies #Algorithms and Data Compression #Computation and Language (cs.CL) #FOS: Computer and information sciences #cs.CL

paper · pdf · doi:10.48550/arxiv.2405.12532

openalex publication_date 2024/05/21 · arxiv published 2024/05/21 · openalex created_date 2024/05/23 · arxiv updated 2024/06/05 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) have shown remarkable comprehension abilities but face challenges in GPU memory usage during inference, hindering their scalability for real-time applications like chatbots. To accelerate inference, we store computed keys and values (KV cache) in the GPU memory. Existing methods study the KV cache compression to reduce memory by pruning the pre-computed KV cache. However, they neglect the inter-layer dependency between layers and huge memory consumption in pre-computation. To explore these deficiencies, we find that the number of crucial keys and values that influence future generations decreases layer by layer and we can extract them by the consistency in attention weights. Based on the findings, we propose PyramidInfer, a method that compresses the KV cache by layer-wise retaining crucial context. PyramidInfer saves significant memory by computing fewer keys and values without sacrificing performance. Experimental results show PyramidInfer improves 2.2x throughput compared to Accelerate with over 54% GPU memory reduction in KV cache.

Cited by

• DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention
• KV Admission: Learning What to Write for Efficient Long-Context Inference
• Mindscape-Aware Retrieval Augmented Generation for Improved Long Context Understanding
• V-Rex: Real-Time Streaming Video LLM Acceleration via Dynamic KV Cache Retrieval
• SJD++: Improved Speculative Jacobi Decoding for Training-free Acceleration of Discrete Auto-regressive Text-to-Image Generation
• Autoregressive Image Generation Needs Only a Few Lines of Cached Tokens
• KVReviver: Reversible KV Cache Compression with Sketch-Based Token Reconstruction
• G-KV: Decoding-Time KV Cache Eviction with Global Attention
• Lethe: Layer- and Time-Adaptive KV Cache Pruning for Reasoning-Intensive LLM Serving
• SALS: Sparse Attention in Latent Space for KV cache Compression
• Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Eviction
• Kelle: Co-design KV Caching and eDRAM for Efficient LLM Serving in Edge Computing
• XQuant: Achieving Ultra-Low Bit KV Cache Quantization with Cross-Layer Compression
• Which Heads Matter for Reasoning? RL-Guided KV Cache Compression
• Artificial Hippocampus Networks for Efficient Long-Context Modeling
• GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awareness
• UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compression
• Judge Q: Trainable Queries for Optimized Information Retention in KV Cache Eviction
• LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocation
• EvolKV: Evolutionary KV Cache Compression for LLM Inference
• PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference
• GraphKV: Breaking the Static Selection Paradigm with Graph-Based KV Cache Eviction
• SparK: Query-Aware Unstructured Sparsity with Recoverable KV Cache Channel Pruning
• AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunities
• P/D-Device: Disaggregated Large Language Model between Cloud and Devices
• KV Cache Compression for Inference Efficiency in LLMs: A Review
• CompressKV: Semantic Retrieval Heads Know What Tokens are Not Important Before Generation
• SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference
• CaliDrop: KV Cache Compression with Calibration
• DAC: A Dynamic Attention-aware Approach for Task-Agnostic Prompt Compression
• IAM: Efficient Inference through Attention Mapping between Different-scale LLMs
• KV-Latent: Dimensional-level KV Cache Reduction with Frequency-aware Rotary Positional Embedding
• LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models
• Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation
• Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMs
• SpindleKV: A Novel KV Cache Reduction Method Balancing Both Shallow and Deep Layers
• ReSum: Synergizing LLM Reasoning and Summarization with Reinforcement Learning
• A Survey of LLM Inference Systems
• Breaking the Boundaries of Long-Context LLM Inference: Adaptive KV Management on a Single Commodity GPU
• CentroidKV: Efficient Long-Context LLM Inference via KV Cache Clustering
• AnchorAttention: Difference-Aware Sparse Attention with Stripe Granularity
• KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction
• FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusion
• Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compression
• Efficient Multi-modal Long Context Learning for Training-free Adaptation
• FlowKV: Enhancing Multi-Turn Conversational Coherence in LLMs via Isolated Key-Value Cache Management
• Fractured Chain-of-Thought Reasoning
• Practical Online KV Cache Compaction for LLM Agents: An Empirical Study
• KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV Cache
• S⁴R: Selective Sampling, Subspaces, and Sparse Reconstruction for Compressed Long-Context KV Caching
• VeriCache: Turning Lossy KV Cache into Lossless LLM Inference
• ProphetKV: User-Query-Driven Selective Recomputation for Efficient KV Cache Reuse in Retrieval-Augmented Generation
• The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs
• KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments
• KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inference

Discussions

• 3. PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference arxiv.org/abs/2405.12532 4. Any-Order Flexible Length Masked Diffusion arxiv.org/abs/2509.01025 [bsky, 0 points, 0 comments]

Related

• PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling
• XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference
• HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading
• KV Cache Compression for Inference Efficiency in LLMs: A Review
• HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs