Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimization

2018/06/20 by Jiaxiang Wu, Wu, Jiaxiang, Weidong Huang +5 · 14 citations
Computer Science · Engineering · #Computer Vision and Pattern Recognition (cs.CV) #Distributed #Distributed and Parallel Computing Systems #FOS: Computer and information sciences #Machine Learning and ELM #Metaheuristic Optimization Algorithms Research #Parallel #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques #and Cluster Computing (cs.DC) #cs.CV #cs.DC

paper · pdf · doi:10.48550/arxiv.1806.08054

Accepted by ICML 2018

arxiv created 2018/06/21 · openalex publication_date 2018/06/21 · arxiv updated 2018/06/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large-scale distributed optimization is of great importance in various applications. For data-parallel based distributed learning, the inter-node gradient communication often becomes the performance bottleneck. In this paper, we propose the error compensated quantized stochastic gradient descent algorithm to improve the training efficiency. Local gradients are quantized to reduce the communication overhead, and accumulated quantization error is utilized to speed up the convergence. Furthermore, we present theoretical analysis on the convergence behaviour, and demonstrate its advantage over competitors. Extensive experiments indicate that our algorithm can compress gradients by a factor of up to two magnitudes without performance degradation.

Cited by

• Machine Learning and CPU (Central Processing Unit) Scheduling Co-Optimization over a Network of Computing Centers
• Local SGD: Unified Theory and New Efficient Methods
• Understanding Top-k Sparsification in Distributed Deep Learning
• Layer-wise Adaptive Gradient Sparsification for Distributed Deep Learning with Convergence Guarantees
• Smoothness Matrices Beat Smoothness Constants: Better Communication Compression Techniques for Distributed Optimization
• Tight analyses of first-order methods with error feedback
• Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism
• It Takes a Good Model to Train a Good Model: Generalized Gaussian Priors for Optimized LLMs
• Election Coding for Distributed Learning: Protecting SignSGD against Byzantine Attacks
• Communication Efficient Federated Learning with Energy Awareness over Wireless Networks
• A Distributed Synchronous SGD Algorithm with Global Top-k Sparsification for Low Bandwidth Networks
• γ-FedHT: Stepsize-Aware Hard-Threshold Gradient Compression in Federated Learning
• A Tight Theory of Error Feedback Algorithms in Distributed Optimization
• Distributed Sparse SGD with Majority Voting

Related

• Truncated Non-Uniform Quantization for Distributed SGD
• Unbiased Single-scale and Multi-scale Quantizers for Distributed Optimization
• DQ-SGD: Dynamic Quantization in SGD for Communication-Efficient Distributed Learning
• Trustworthy Efficient Communication for Distributed Learning using LQ-SGD Algorithm
• Improved Quantization Strategies for Managing Heavy-tailed Gradients in Distributed Learning