2020/07/01 by Xiaorui Liu, Yao Li, Liu, Xiaorui +7 · 4 citations
Computer Science · Engineering · Mathematics · #Distributed #Distributed Control Multi-Agent Systems #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Parallel #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques #and Cluster Computing (cs.DC) #cs.DC #cs.LG #stat.ML
paper · pdf · doi:10.48550/arxiv.2007.00232
ICLR 2021 (International Conference on Learning Representations)
openalex publication_date 2020/07/01 · arxiv created 2021/03/18 · arxiv updated 2021/03/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Communication compression has become a key strategy to speed up distributed optimization. However, existing decentralized algorithms with compression mainly focus on compressing DGD-type algorithms. They are unsatisfactory in terms of convergence rate, stability, and the capability to handle heterogeneous data. Motivated by primal-dual algorithms, this paper proposes the first \underlineLin\underlineEAr convergent \underlineDecentralized algorithm with compression, LEAD. Our theory describes the coupled dynamics of the inexact primal and dual update as well as compression error, and we provide the first consensus error bound in such settings without assuming bounded gradients. Experiments on convex problems validate our theoretical analysis, and empirical study on deep neural nets shows that LEAD is applicable to non-convex problems.