Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving

2024/06/24 by Ruoyu Qin, Qin, Ruoyu, Zheming Li +11 · 3 voices · 33 citations
Engineering · #Plasma Diagnostics and Applications #Radiation Effects in Electronics #Semiconductor materials and devices #cs.AI #cs.AR #cs.DC

paper · pdf · doi:10.48550/arxiv.2407.00079

openalex publication_date 2024/06/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI. It features a KVCache-centric disaggregated architecture that separates the prefill and decoding clusters. It also leverages the underutilized CPU, DRAM, and SSD resources of the GPU cluster to implement a disaggregated cache of KVCache. The core of Mooncake is its KVCache-centric scheduler, which balances maximizing overall effective throughput while meeting latency-related Service Level Objectives (SLOs). Unlike traditional studies that assume all requests will be processed, Mooncake faces challenges due to highly overloaded scenarios. To mitigate these, we developed a prediction-based early rejection policy. Experiments show that Mooncake excels in long-context scenarios. Compared to the baseline method, Mooncake can achieve up to a 525% increase in throughput in certain simulated scenarios while adhering to SLOs. Under real workloads, Mooncake's innovative architecture enables Kimi to handle 75% more requests.

Cited by

• ExpertPlex: A High-Goodput Disaggregated Serving System for MoE LLMs with Adaptive Persistent Kernels
• DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training
• Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacenter
• Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model
• MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoning
• Enabling Disaggregated Multi-Stage MLLM Inference via GPU-Internal Scheduling and Resource Sharing
• ProServe: Unified Multi-Priority Request Scheduling for LLM Serving
• TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity
• OOCO: Latency-disaggregated Architecture for Online-Offline Co-locate LLM Serving
• OmniInfer: System-Wide Acceleration Techniques for Optimizing LLM Serving Throughput and Latency
• On 10x Better Scalability: KV Stores Scale Up KV Cache
• Understanding Inference Scaling for LLMs: Bottlenecks, Trade-offs, and Performance Principles
• From Models to Operators: Rethinking Autoscaling Granularity for Large Generative Models
• Tree Training: Accelerating Agentic LLMs Training via Shared Prefix Reuse
• AsyncHZP: Hierarchical ZeRO Parallelism with Asynchronous Scheduling for Scalable LLM Training
• StreamingThinker: Large Language Models Can Think While Reading
• BanaServe: Unified KV Cache and Dynamic Module Migration for Balancing Disaggregated LLM Serving in AI Infrastructure
• On the Role of Temperature Sampling in Test-Time Scaling
• LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
• From Principles to Practice: A Systematic Study of LLM Serving on Multi-core NPUs
• Cache-to-Cache: Direct Semantic Communication Between Large Language Models
• SemShareKV: Efficient KVCache Sharing for Semantically Similar Prompts via Token-Level LSH Matching
• GORGO: Online Tuning for Cross-Region Network-Aware LLM Serving
• Disaggregated Prefill and Decoding Inference System for Large Language Model Serving on Multi-Vendor GPUs
• Expert-as-a-Service: Towards Efficient, Scalable, and Robust Large-scale MoE Serving
• Shift Parallelism: Low-Latency, High-Throughput LLM Inference for Dynamic Workloads
• PDTrim: Targeted Pruning for Prefill-Decode Disaggregation in Inference
• HyperFlexis: Joint Design of Algorithms and Systems for Multi-SLO Serving and Fast Scaling
• Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inference
• Efficient Speculative Decoding for Llama at Scale: Challenges and Solutions
• Block: Balancing Load in LLM Serving with Context, Knowledge and Predictive Scheduling
• FlashCommunication V2: Bit Splitting and Spike Reserving for Any Bit Communication
• Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving

Discussions

• MoonshotAI unveils Kimi's large-scale LLM serving architecture [hn, 18 points, 1 comments]
• This paper explains in great detail, Glenn. arxiv.org/pdf/2407.00079 [bsky, 3 points, 1 comments]
• arxiv 📄 Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving http://arxiv.org/abs/2407.00079v3 Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonsho [bsky, 0 points, 0 comments]

Related

• TENT: A Declarative Slice Spraying Engine for Performant and Resilient Data Movement in Disaggregated LLM Serving
• FastCache: Optimizing Multimodal LLM Serving through Lightweight KV-Cache Compression Framework
• Tokencake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications
• Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
• EconoServe: Maximizing Multi-Resource Utilization with SLO Guarantees in LLM Serving