2024/02/02 by Guangfeng Yan, Li Tan, Yan, Guangfeng +7
Computer Science · Engineering · #Distributed #Distributed and Parallel Computing Systems #Experimental Learning in Engineering #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural Networks and Applications #Parallel #and Cluster Computing (cs.DC)
paper · pdf · doi:10.48550/arxiv.2402.01798
openalex publication_date 2024/02/02 · openalex created_date 2024/02/07 · openalex updated_date 2026/07/28
Gradient compression has surfaced as a key technique to address the challenge of communication efficiency in distributed learning. In distributed deep learning, however, it is observed that gradient distributions are heavy-tailed, with outliers significantly influencing the design of compression strategies. Existing parameter quantization methods experience performance degradation when this heavy-tailed feature is ignored. In this paper, we introduce a novel compression scheme specifically engineered for heavy-tailed gradients, which effectively combines gradient truncation with quantization. This scheme is adeptly implemented within a communication-limited distributed Stochastic Gradient Descent (SGD) framework. We consider a general family of heavy-tail gradients that follow a power-law distribution, we aim to minimize the error resulting from quantization, thereby determining optimal values for two critical parameters: the truncation threshold and the quantization density. We provide a theoretical analysis on the convergence error bound under both uniform and non-uniform quantization scenarios. Comparative experiments with other benchmarks demonstrate the effectiveness of our proposed method in managing the heavy-tailed gradients in a distributed learning environment.