SiPipe: Bridging the CPU-GPU Utilization Gap for Efficient Pipeline-Parallel LLM Inference

2025/06/27 by Yun He, He, Yongchao, Zheng Cao +2 · 1 citation
Computer Science · #Advanced Neural Network Applications #Big Data and Digital Economy #Distributed #FOS: Computer and information sciences #Parallel #Parallel Computing and Optimization Techniques #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2506.22033

openalex publication_date 2025/06/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

As inference workloads for large language models (LLMs) scale to meet growing user demand, pipeline parallelism (PP) has become a widely adopted strategy for multi-GPU deployment, particularly in cross-node setups, to improve key-value (KV) cache capacity and inference throughput. However, PP suffers from inherent inefficiencies caused by three types of execution bubbles-load-imbalance, intra-stage, and inter-stage-which limit pipeline saturation. We present SiPipe, a heterogeneous pipeline design that improves throughput by leveraging underutilized CPU resources to offload auxiliary computation and communication. SiPipe incorporates three key techniques-CPU sampling, a token-safe execution model, and structure-aware transmission-to mitigate pipeline bubbles and improve execution efficiency. Across diverse LLMs, SiPipe achieves up to 2.1 times higher throughput, 43% lower per-token latency, and up to 23% higher average GPU utilization compared to the state-of-the-art vLLM under the same PP configuration, demonstrating its generality across LLMs and deployment scenarios.

Citations

• gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttling
• PipeLLM: Fast and Confidential Large Language Model Services with Speculative Pipelined Encryption
• The Llama 3 Herd of Models
• PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculation
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
• SGLang: Efficient Execution of Structured Language Model Programs
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Pipeline MoE: A Flexible MoE Implementation with Pipeline Parallelism
• Efficient Training of Language Models to Fill in the Middle
• Training language models to follow instructions with human feedback
• The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink
• Language Models are Few-Shot Learners
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• The Curious Case of Neural Text Degeneration
• SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing
• Hierarchical Neural Story Generation
• Attention Is All You Need
• Deep Residual Learning for Image Recognition
• Neural Machine Translation of Rare Words with Subword Units
• PipeFusion: Patch-level Pipeline Parallelism for Diffusion Transformers Inference

Cited by

• A Unified Sparse Attention via Multi-Granularity Compression

Related

• PIPO: Pipelined Offloading for Efficient Inference on Consumer Devices
• TD-Pipe: Temporally-Disaggregated Pipeline Parallelism Architecture for High-Throughput LLM Inference
• SIMPLE: Disaggregating Sampling from GPU Inference into a Decision Plane for Faster Distributed LLM Serving
• SpecPipe: Accelerating Pipeline Parallelism-based LLM Inference with Speculative Decoding
• gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttling