PiKV: KV Cache Management System for Mixture of Experts

2025/08/02 by Dong Liu, Liu, Dong, Yanxuan Yu +6 · 4 citations
Computer Science · #Parallel Computing and Optimization Techniques #Cloud Computing and Resource Management #Caching and Content Delivery

paper · pdf · doi:10.48550/arxiv.2508.06526

Abstract

As large-scale language models continue to scale up in both size and context length, the memory and communication cost of key-value (KV) cache storage has become a major bottleneck in multi-GPU and multi-node inference. While MoE-based architectures sparsify computation across experts, the corresponding KV caches remain dense and globally synchronized, resulting in significant overhead. We introduce PiKV, a parallel and distributed KV cache serving framework tailored for MoE architecture. PiKV leverages expert-sharded KV storage to partition caches across GPUs, PiKV routing to reduce token-to-KV access, and a PiKV Scheduling to adaptively retain query-relevant entries. To further reduce memory usage, PiKV integrates PiKV Compression modules the caching pipeline for acceleration. PiKV is recently publicly available as an open-source software library: \hrefhttps://github.com/NoakLiu/PiKVhttps://github.com/NoakLiu/PiKV. PiKV is still a living project, aiming to become a comprehesive KV Cache management system for MoE Architectures.

Citations

• CXL-SpecKV: A Disaggregated FPGA Speculative KV-Cache for Datacenter LLM Serving
• HSGM: Hierarchical Segment-Graph Memory for Scalable Long-Text Semantics
• TinyServe: Query-Aware Cache Selection for Efficient LLM Serving
• FastCache: Fast Caching for Diffusion Transformer Through Learnable Linear Approximation
• Qwen2.5-VL Technical Report
• Designing Large Foundation Models for Efficient Training and Inference: A Survey
• LLMEasyQuant: Scalable Quantization for Parallel and Distributed LLM Inference
• MT2ST: Adaptive Multi-Task to Single-Task Learning
• Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
• PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling
• HMT: Hierarchical Memory Transformer for Efficient Long Context Language Processing
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
• Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention
• An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models
• Training-Free Long-Context Scaling of Large Language Models
• Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
• Ring Attention with Blockwise Transformers for Near-Infinite Context
• Efficient Streaming Language Models with Attention Sinks
• YaRN: Efficient Context Window Extension of Large Language Models
• Extending Context Window of Large Language Models via Positional Interpolation
• H₂O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
• FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
• PIT: Optimization of Dynamic Sparse Deep Learning Models via Permutation Invariant Transformation
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
• LoRA: Low-Rank Adaptation of Large Language Models
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Big Bird: Transformers for Longer Sequences
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Longformer: The Long-Document Transformer
• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
• Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

Cited by

• CSV-Decode: Certifiable Sub-Vocabulary Decoding for Efficient Large Language Model Inference
• Cognitive Load Traces as Symbolic and Visual Accounts of Deep Model Cognition
• TinyServe: Query-Aware Cache Selection for Efficient LLM Serving
• QuickMerge++: Fast Token Merging with Autoregressive Prior

Related

• DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction
• KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing
• LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
• KV-CAR: KV Cache Compression using Autoencoders and KV Reuse in Large Language Models
• EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Serving