2020/06/30 by Nir Shlezinger, Mingzhe Chen, Yonina C. Eldar +2
Computer Science · Engineering · Mathematics · #Algorithm #Artificial intelligence #Computer hardware #Computer science #Digital signal processing #Learning vector quantization #Privacy-Preserving Technologies in Data #Quantization (signal processing) #Signal processing #Stochastic Gradient Optimization Techniques #Theoretical computer science #Vector quantization #Wireless Communication Security Techniques #cs.IT #cs.LG #math.IT #stat.ML
paper · pdf · doi:10.1109/tsp.2020.3046971
arxiv created 2020/12/14 · openalex publication_date 2020/12/23 · arxiv updated 2021/02/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06
Traditional deep learning models are trained at a centralized server using data samples collected from users. Such data samples often include private information, which the users may not be willing to share. Federated learning (FL) is an emerging approach to train such learning models without requiring the users to share their data. FL consists of an iterative procedure, where in each iteration the users train a copy of the learning model locally. The server then collects the individual updates and aggregates them into a global model. A major challenge that arises in this method is the need of each user to repeatedly transmit its learned model over the throughput limited uplink channel. In this work, we tackle this challenge using tools from quantization theory. In particular, we identify the unique characteristics associated with conveying trained models over rate-constrained channels, and propose a suitable quantization scheme for such settings, referred to as universal vector quantization for FL (UVeQFed). We show that combining universal vector quantization methods with FL yields a decentralized training system in which the compression of the trained models induces only a minimum distortion. We then theoretically analyze the distortion, showing that it vanishes as the number of users grows. We also characterize how models trained with conventional federated averaging combined with UVeQFed converge to the model which minimizes the loss function. Our numerical results demonstrate the gains of UVeQFed over previously proposed methods in terms of both distortion induced in quantization and accuracy of the resulting aggregated model.