Eigen Attention: Attention in Low-Rank Space for KV Cache Compression

2024/08/10 by Utkarsh Saxena, Gobinda Saha, Saxena, Utkarsh +5 · 29 citations
Computer Science · Mathematics · #Advanced Data Compression Techniques #Cache #Combinatorics #Compression (physics) #Computer science #Distributed and Parallel Computing Systems #Materials science #Mathematics #Operating system #Parallel Computing and Optimization Techniques #Parallel computing #Rank (graph theory) #Space (punctuation)

paper · pdf · doi:10.48550/arxiv.2408.05646

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/08/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large language models (LLMs) represent a groundbreaking advancement in the domain of natural language processing due to their impressive reasoning abilities. Recently, there has been considerable interest in increasing the context lengths for these models to enhance their applicability to complex tasks. However, at long context lengths and large batch sizes, the key-value (KV) cache, which stores the attention keys and values, emerges as the new bottleneck in memory usage during inference. To address this, we propose Eigen Attention, which performs the attention operation in a low-rank space, thereby reducing the KV cache memory overhead. Our proposed approach is orthogonal to existing KV cache compression techniques and can be used synergistically with them. Through extensive experiments over OPT, MPT, and Llama model families, we demonstrate that Eigen Attention results in up to 40% reduction in KV cache sizes and up to 60% reduction in attention operation latency with minimal drop in performance. Code is available at https://github.com/UtkarshSaxena1/EigenAttn.

Cited by

• DynaCalKV: Key-Value Cache Compression via Head Grouping and Adaptive Rank Allocation
• LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decoding
• MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoning
• KV Admission: Learning What to Write for Efficient Long-Context Inference
• KQ-SVD: Compressing the KV Cache with Provable Guarantees on Attention Fidelity
• Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs
• SWAN: Sparse Winnowed Attention for Reduced Inference Memory via Decompression-Free KV-Cache Compression
• KV Cache Transform Coding for Compact Storage in LLM Inference
• SALS: Sparse Attention in Latent Space for KV cache Compression
• NOSA: Native and Offloadable Sparse Attention
• KaVa: Latent Reasoning via Compressed KV-Cache Distillation
• KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correction
• OjaKV: Context-Aware Online Low-Rank KV Cache Compression with Oja's Rule
• AQUA: Attention via QUery mAgnitudes for Memory and Compute Efficient Inference in LLMs
• XQuant: Breaking the Memory Wall for LLM Inference with KV Cache Rematerialization
• SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Serving
• KV-Latent: Dimensional-level KV Cache Reduction with Frequency-aware Rotary Positional Embedding
• Cartridges: Lightweight and general-purpose long context representations via self-study
• Inference-Time Hyper-Scaling with KV Cache Compression
• R-KV: Redundancy-aware KV Cache Compression for Reasoning Models
• ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibration
• Hardware-Efficient Attention for Fast Decoding
• ASTRA: Communication-Efficient Acceleration for Multi-Device Transformer Inference
• μ-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Experts
• LatentLLM: Attention-Aware Joint Tensor Compression
• FlowKV: Enhancing Multi-Turn Conversational Coherence in LLMs via Isolated Key-Value Cache Management
• S⁴R: Selective Sampling, Subspaces, and Sparse Reconstruction for Compressed Long-Context KV Caching
• FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference
• SAKI: Score-Aware Low-Rank Key Indexing with Random-Matrix Noise Correction for KV Retrieval

Related

• DBudgetKV: Dynamic Budget in KV Cache Compression for Ensuring Optimal Performance
• Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
• Beyond KV Caching: Shared Attention for Efficient LLMs
• A Simple and Effective L₂ Norm-Based Strategy for KV Cache Compression
• Effectively Compress KV Heads for LLM