Blink: Fast and Generic Collectives for Distributed ML

2019/10/11 by Guanhua Wang, Shivaram Venkataraman, Wang, Guanhua +9 · 8 citations
Computer Science · #Advanced Neural Network Applications #Distributed #FOS: Computer and information sciences #Machine Learning (cs.LG) #Parallel #Parallel Computing and Optimization Techniques #Stochastic Gradient Optimization Techniques #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.1910.04940

openalex publication_date 2019/10/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Model parameter synchronization across GPUs introduces high overheads for data-parallel training at scale. Existing parameter synchronization protocols cannot effectively leverage available network resources in the face of ever increasing hardware heterogeneity. To address this, we propose Blink, a collective communication library that dynamically generates optimal communication primitives by packing spanning trees. We propose techniques to minimize the number of trees generated and extend Blink to leverage heterogeneous communication channels for faster data transfers. Evaluations show that compared to the state-of-the-art (NCCL), Blink can achieve up to 8x faster model synchronization, and reduce end-to-end training time for image classification tasks by up to 40%.

Cited by

• Enabling Reconfiguration-Communication Overlap for Collective Communication in Optical Networks
• Short-circuiting Rings for Low-Latency AllReduce
• An Efficient, Reliable and Observable Collective Communication Library in Large-scale GPU Training Clusters
• Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
• PCCL: Photonic circuit-switched collective communication for distributed ML
• Characterizing the Efficiency of Distributed Training: A Power, Performance, and Thermal Perspective
• Hoplite: Efficient and Fault-Tolerant Collective Communication for Task-Based Distributed Systems
• Synthesizing Optimal Parallelism Placement and Reduction Strategies on Hierarchical Systems for Deep Learning

Related

• GC3: An Optimizing Compiler for GPU Collective Communication
• Optimizing Distributed ML Communication with Fused Computation-Collective Operations
• The Big Send-off: High Performance Collectives on GPU-based Supercomputers
• TACCL: Guiding Collective Algorithm Synthesis using Communication Sketches
• PCCL: Photonic circuit-switched collective communication for distributed ML