2020/11/12 by Kerem Özfatura, Kerem Ozfatura, Ozfatura, Kerem +6 · 1 citation
Computer Science · Mathematics · #Algorithm #Artificial intelligence #Artificial neural network #Bottleneck #Communication complexity #Computer science #Distributed #Distributed computing #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Federated learning #Information Theory (cs.IT) #Machine Learning (cs.LG) #Majority rule #Mathematics #Parallel #Privacy-Preserving Technologies in Data #Reduction (mathematics) #Representation (politics) #Sparse approximation #Stochastic Gradient Optimization Techniques #Stochastic gradient descent #Theoretical computer science #Voting #and Cluster Computing (cs.DC) #cs.DC #cs.IT #cs.LG #math.IT
paper · pdf · doi:10.48550/arxiv.2011.06495
published in arXiv (Cornell University) (Cornell University)
arxiv created 2020/11/12 · openalex publication_date 2020/11/12 · arxiv updated 2020/11/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Distributed learning, particularly variants of distributed stochastic gradient descent (DSGD), are widely employed to speed up training by leveraging computational resources of several workers. However, in practise, communication delay becomes a bottleneck due to the significant amount of information that needs to be exchanged between the workers and the parameter server. One of the most efficient strategies to mitigate the communication bottleneck is top-K sparsification. However, top-K sparsification requires additional communication load to represent the sparsity pattern, and the mismatch between the sparsity patterns of the workers prevents exploitation of efficient communication protocols. To address these issues, we introduce a novel majority voting based sparse communication strategy, in which the workers first seek a consensus on the structure of the sparse representation. This strategy provides a significant reduction in the communication load and allows using the same sparsity level in both communication directions. Through extensive simulations on the CIFAR-10 dataset, we show that it is possible to achieve up to x4000 compression without any loss in the test accuracy.