Sandwich: Joint Configuration Search and Hot-Switching for Efficient CPU LLM Serving

2025/05/19 by Juntao Zhao, J. Li, Zhao, Juntao +3 · 1 citation
Computer Science · #Algorithms and Data Compression #Artificial Intelligence (cs.AI) #Distributed #FOS: Computer and information sciences #Hardware Architecture (cs.AR) #Network Packet Processing and Optimization #Parallel #Programming Languages (cs.PL) #VLSI and Analog Circuit Testing #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2507.18454

openalex publication_date 2025/05/19 · openalex created_date 2025/10/07 · openalex updated_date 2026/07/28

Abstract

CPUs are critical for LLM serving due to their availability, cost efficiency, and edge applicability. However, efficient CPU serving is hindered by conflicting prefill/decode resource demands under non-disaggregated deployment constraints--existing solutions fail to avoid cross-phase interference, ignore sub-NUMA hardware structures, and deliver suboptimal dynamic-shape kernel performance. We propose Sandwich, a full-stack CPU LLM serving system with three core innovations addressing these challenges: (1) seamless phase-wise plan switching to eliminate cross-phase interference; (2) TopoTree, a tree-based hardware abstraction for automated substructure-aware (e.g., LLC slices) partial core allocation; (3) fast-start-then-finetune dynamic-shape tensor program generation. Across five x86/ARM CPU platforms, Sandwich achieves an average 2.01x end-to-end speedup and up to 3.40x latency reduction over state-of-the-art systems. Its kernels match static compiler performance with three orders of magnitude lower tuning cost.

Citations

• Small Language Models are the Future of Agentic AI
• On-Device Language Models: A Comprehensive Review
• Characterization of Large Language Model Development in the Datacenter
• LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• CASSINI: Network-Aware Job Scheduling in Machine Learning Clusters
• LLaMA: Open and Efficient Foundation Language Models
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Ansor: Generating High-Performance Tensor Programs for Deep Learning
• Learning to Optimize Tensor Programs
• Hoard

Cited by

• TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference

Related

• SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
• SPAD: Specialized Prefill and Decode Hardware for Disaggregated LLM Inference
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
• Highly Optimized Kernels and Fine-Grained Codebooks for LLM Inference on Arm CPUs
• Efficient LLM Inference on CPUs