Breadth-First Pipeline Parallelism

2022/11/11 by Joël Lamy-Poirier, Lamy-Poirier, Joel · 5 citations
Computer Science · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Distributed #FOS: Computer and information sciences #Machine Learning (cs.LG) #Parallel #Parallel Computing and Optimization Techniques #Stochastic Gradient Optimization Techniques #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2211.05953

openalex publication_date 2022/11/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce Breadth-First Pipeline Parallelism, a novel training schedule which optimizes the combination of pipeline and data parallelism. Breadth-First Pipeline Parallelism lowers training time, cost and memory usage by combining a high GPU utilization with a small batch size per GPU, and by making use of fully sharded data parallelism. Experimentally, we observed an increase of up to 43% in training throughput for a 52 billion-parameter model using a small batch size per GPU compared to Megatron-LM, which would reduce the training time and cost by the same amount on a large GPU cluster.

Cited by

• A Scalable Distributed Framework for Multimodal GigaVoxel Image Registration
• AdaPtis: Reducing Pipeline Bubbles with Adaptive Pipeline Parallelism on Heterogeneous Models
• Kimi K2: Open Agentic Intelligence
• Zorse: Optimizing LLM Training Efficiency on Heterogeneous GPU Clusters
• Compute Requirements for Algorithmic Innovation in Frontier AI Models

Related

• DAPPLE: A Pipelined Data Parallel Approach for Training Large Models
• FreeRide: Harvesting Bubbles in Pipeline Parallelism
• DawnPiper: A Memory-scablable Pipeline Parallel Training Framework
• Zero Bubble Pipeline Parallelism
• Seq1F1B: Efficient Sequence-Level Pipeline Parallelism for Large Language Model Training