RAP: KV-Cache Compression via RoPE-Aligned Pruning

2026/02/28 by Jihao Xin, Tian Lyu, David Keyes +2
Computer Science · #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.2602.02599

arxiv created 2026/07/30 · arxiv updated 2026/08/04

Abstract

Long-context inference in large language models (LLMs) is bottlenecked by the memory and compute of the key-value (KV) cache. Structured pruning is a direct way to shrink it: dropping the least useful channels of the W_k, W_v projection weights to reduce the output KV dimensions. However, modern LLMs apply Rotary Position Embedding (RoPE) after the QK projections, which rotates feature dimensions in pairs. Therefore, removing individual channels breaks these pairs, corrupting RoPE's positional semantics and rendering the pruned model unusable. We propose RoPE-Aligned Pruning (RAP), which constrains the pruning granularity to RoPE-aligned pairs rather than individual channels: removing whole pairs to keep the rotation intact. Our evaluation across Llama, Mistral, and Qwen models from 3B to 14B shows that RAP preserves accuracy at 30% KV compression (retain ratio \rho = 0.7), far outperforms RoPE-blind channel pruning, stays near the strongest low-rank method at lower attention cost, and composes with orthogonal methods such as quantization.

Citations

• Lethe: Layer- and Time-Adaptive KV Cache Pruning for Reasoning-Intensive LLM Serving
• SparK: Query-Aware Unstructured Sparsity with Recoverable KV Cache Channel Pruning
• LeanK: Learnable K Cache Channel Pruning for Efficient Decoding
• ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibration
• KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction
• Does quantization affect models' performance on long-context tasks?
• Circle-RoPE: Cone-like Decoupled Rotary Positional Embedding for Large Vision-Language Models
• EliteKV: Scalable KV Cache Compression via RoPE Frequency Selection and Joint Low-Rank Projection
• The Rotary Position Embedding May Cause Dimension Inefficiency in Attention Heads for Long-Distance Retrieval
• SCBench: A KV Cache-Centric Analysis of Long-Context Methods
• MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projection
• Palu: Compressing KV-Cache with Low-Rank Projection
• ThinK: Thinner Key Cache by Query-Driven Pruning
• Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysis
• QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving
• SnapKV: LLM Knows What You are Looking for Before Generation
• AI and Memory Wall
• SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression
• KIVI : Plug-and-play 2bit KV Cache Quantization with Streaming Asymmetric Quantization
• KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
• Qwen Technical Report
• YaRN: Efficient Context Window Extension of Large Language Models
• LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
• H₂O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
• A Simple and Effective Pruning Approach for Large Language Models
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
• LLM-Pruner: On the Structural Pruning of Large Language Models
• GPT-4 Technical Report
• LLaMA: Open and Efficient Foundation Language Models
• SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
• A Fast Post-Training Pruning Framework for Transformers
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Big Bird: Transformers for Longer Sequences
• Longformer: The Long-Document Transformer
• PIQA: Reasoning about Physical Commonsense in Natural Language
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Importance Estimation for Neural Network Pruning
• HellaSwag: Can a Machine Really Finish Your Sentence?
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Attention Is All You Need
• Pointer Sentinel Mixture Models
• Distilling the Knowledge in a Neural Network
• The Approximation of One Matrix by Another of Lower Rank