XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN Training

2019/10/24 by Lei Guan, Guan, Lei, Wotao Yin +5 · 2 citations
Computer Science · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and ELM #Performance (cs.PF)

paper · pdf · doi:10.48550/arxiv.1911.04610

openalex publication_date 2019/10/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose XPipe, an efficient asynchronous pipeline model parallelism approach for multi-GPU DNN training. XPipe is designed to use multiple GPUs to concurrently and continuously train different parts of a DNN model. To improve GPU utilization and achieve high throughput, it splits a mini-batch into a set of micro-batches. It allows the overlapping of the pipelines of multiple micro-batches, including those belonging to different mini-batches. Most importantly, the novel weight prediction strategy adopted by XPipe enables it to effectively address the weight inconsistency and staleness issues incurred by the asynchronous pipeline parallelism. As a result, XPipe incorporates the advantages of both synchronous and asynchronous pipeline model parallelism approaches. Concretely, it can achieve very comparable (even slightly better) model accuracy as its synchronous counterpart while obtaining higher throughput than it. Experimental results show that XPipe outperforms other state-of-the-art synchronous and asynchronous model parallelism approaches.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Device Placement Optimization with Reinforcement Learning
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• PipeDream: Fast and Efficient Pipeline Parallel DNN Training
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Automatic Graph Partitioning for Very Large-scale Deep Learning
• torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Models

Related

• Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform
• DAPPLE: A Pipelined Data Parallel Approach for Training Large Models
• HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU\n Clusters through Integration of Pipelined Model Parallelism and Data\n Parallelism
• PipeMare: Asynchronous Pipeline Parallel DNN Training
• PipeDream: Fast and Efficient Pipeline Parallel DNN Training