2020/03/13 by Sarit Khirirat, Sindri Magnússon, Khirirat, Sarit +5
Computer Science · Mathematics · #Data Stream Mining Techniques #FOS: Computer and information sciences #FOS: Mathematics #IoT and Edge/Fog Computing #Machine Learning (stat.ML) #Optimization and Control (math.OC) #Stochastic Gradient Optimization Techniques #math.OC #stat.ML
paper · pdf · doi:10.48550/arxiv.2003.06377
27 pages, 11 figures, 1 table
openalex publication_date 2020/03/13 · arxiv created 2020/06/17 · arxiv updated 2020/06/18 · openalex created_date 2020/06/25 · openalex updated_date 2026/07/28
With the increasing scale of machine learning tasks, it has become essential to reduce the communication between computing nodes. Early work on gradient compression focused on the bottleneck between CPUs and GPUs, but communication-efficiency is now needed in a variety of different system architectures, from high-performance clusters to energy-constrained IoT devices. In the current practice, compression levels are typically chosen before training and settings that work well for one task may be vastly suboptimal for another dataset on another architecture. In this paper, we propose a flexible framework which adapts the compression level to the true gradient at each iteration, maximizing the improvement in the objective function that is achieved per communicated bit. Our framework is easy to adapt from one technology to the next by modeling how the communication cost depends on the compression level for the specific technology. Theoretical results and practical experiments indicate that the automatic tuning strategies significantly increase communication efficiency on several state-of-the-art compression schemes.