Quantized Adam with Error Feedback

2020/04/29 by Congliang Chen, Chen, Congliang, Li Shen +5 · 3 citations
Computer Science · Engineering · #Advanced Memory and Neural Computing #Distributed #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Optimization and Control (math.OC) #Parallel #Stochastic Gradient Optimization Techniques #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2004.14180

openalex publication_date 2020/04/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we present a distributed variant of adaptive stochastic gradient method for training deep neural networks in the parameter-server model. To reduce the communication cost among the workers and server, we incorporate two types of quantization schemes, i.e., gradient quantization and weight quantization, into the proposed distributed Adam. Besides, to reduce the bias introduced by quantization operations, we propose an error-feedback technique to compensate for the quantized gradient. Theoretically, in the stochastic nonconvex setting, we show that the distributed adaptive gradient method with gradient quantization and error-feedback converges to the first-order stationary point, and that the distributed adaptive gradient method with weight quantization and error-feedback converges to the point related to the quantized level under both the single-worker and multi-worker modes. At last, we apply the proposed distributed adaptive gradient methods to train deep neural networks. Experimental results demonstrate the efficacy of our methods.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Gradient-based learning applied to document recognition
• ImageNet classification with deep convolutional neural networks
• On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizes
• Gradient Descent with Compressed Iterates

Cited by

• CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training
• Compressed Decentralized Momentum Stochastic Gradient Methods for Nonconvex Optimization
• Local AdaGrad-Type Algorithm for Stochastic Convex-Concave Optimization

Related

• Efficient-Adam: Communication-Efficient Distributed Adam
• Adaptive Quantization of Model Updates for Communication-Efficient Federated Learning
• Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks
• MQGrad: Reinforcement Learning of Gradient Quantization in Parameter Server
• Towards Efficient Training for Neural Network Quantization