Cooperative SGD: A unified Framework for the Design and Analysis of\n Communication-Efficient SGD Algorithms

2018/08/22 by Jianyu Wang, Gauri Joshi, Wang, Jianyu +1 · 4 citations
Computer Science · Engineering · #Advanced Data Compression Techniques #Distributed #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Medical Image Segmentation Techniques #Parallel #Privacy-Preserving Technologies in Data #Simulation and Modeling Applications #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.1808.07576

openalex publication_date 2018/08/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Communication-efficient SGD algorithms, which allow nodes to perform local\nupdates and periodically synchronize local models, are highly effective in\nimproving the speed and scalability of distributed SGD. However, a rigorous\nconvergence analysis and comparative study of different communication-reduction\nstrategies remains a largely open problem. This paper presents a unified\nframework called Cooperative SGD that subsumes existing communication-efficient\nSGD algorithms such as periodic-averaging, elastic-averaging and decentralized\nSGD. By analyzing Cooperative SGD, we provide novel convergence guarantees for\nexisting algorithms. Moreover, this framework enables us to design new\ncommunication-efficient SGD algorithms that strike the best balance between\nreducing communication overhead and achieving fast error convergence with low\nerror floor.\n

Citations

• Experiments on Parallel Training of Deep Neural Network using Model Averaging
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Parallel SGD: When does averaging help?
• On Nonconvex Decentralized Gradient Descent
• YellowFin and the Art of Momentum Tuning
• Local SGD Converges Fast and Communicates Little
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Learning Performance Optimization for Edge AI System with Time and Energy Constraints
• Decentralized Federated Learning with Distributed Aggregation Weight Optimization
• Communication-Efficient Local Decentralized SGD Methods
• A Hybrid-Order Distributed SGD Method for Non-Convex Optimization to\n Balance Communication Overhead, Computational Complexity, and Convergence\n Rate

Related

• Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization
• Communication-Efficient Local Decentralized SGD Methods
• Asynchronous SGD on Graphs: a Unified Framework for Asynchronous Decentralized and Federated Optimization
• Communication trade-offs for synchronized distributed SGD with large step size
• A Unified Theory of Decentralized SGD with Changing Topology and Local Updates