2019/05/22 by Jingjing Zhang, Osvaldo Simeone, Zhang, Jingjing +1
Computer Science · Engineering · #FOS: Computer and information sciences #FOS: Electrical engineering #Information Theory (cs.IT) #Machine Learning (cs.LG) #Privacy-Preserving Technologies in Data #Signal Processing (eess.SP) #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques #electronic engineering #information engineering
paper · pdf · doi:10.48550/arxiv.1905.09148
openalex publication_date 2019/05/22 · openalex created_date 2019/05/29 · openalex updated_date 2026/07/28
Gradient-based distributed learning in Parameter Server (PS) computing architectures is subject to random delays due to straggling worker nodes, as well as to possible communication bottlenecks between PS and workers. Solutions have been recently proposed to separately address these impairments based on the ideas of gradient coding, worker grouping, and adaptive worker selection. This paper provides a unified analysis of these techniques in terms of wall-clock time, communication, and computation complexity measures. Furthermore, in order to combine the benefits of gradient coding and grouping in terms of robustness to stragglers with the communication and computation load gains of adaptive selection, novel strategies, named Lazily Aggregated Gradient Coding (LAGC) and Grouped-LAG (G-LAG), are introduced. Analysis and results show that G-LAG provides the best wall-clock time and communication performance, while maintaining a low computational cost, for two representative distributions of the computing times of the worker nodes.