Convert, compress, correct: Three steps toward communication-efficient DNN training

2022/03/17 by Chen, Zhong-Jing, Hernandez, Eduin E., Huang, Yu-Chih +1
#FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2203.09044

Abstract

In this paper, we introduce a novel algorithm, CO₃, for communication-efficiency distributed Deep Neural Network (DNN) training. CO₃ is a joint training/communication protocol, which encompasses three processing steps for the network gradients: (i) quantization through floating-point conversion, (ii) lossless compression, and (iii) error correction. These three components are crucial in the implementation of distributed DNN training over rate-constrained links. The interplay of these three steps in processing the DNN gradients is carefully balanced to yield a robust and high-performance scheme. The performance of the proposed scheme is investigated through numerical evaluations over CIFAR-10.

Related

• How to Attain Communication-Efficient DNN Training? Convert, Compress, Correct
• DNN gradient lossless compression: Can GenNorm be the answer?
• ScaleCom: Scalable Sparsified Gradient Compression for Communication-Efficient Distributed Training
• Reducing Data Motion to Accelerate the Training of Deep Neural Networks
• Accelerating Distributed Deep Learning using Lossless Homomorphic Compression