OCTOPUS: Overcoming Performance and Privatization Bottlenecks in Distributed Learning

2021/05/31 by Shuo Wang, Surya Nepal, ‪Surya Nepal‬ +4 · 6 citations
Computer Science · #Advanced Neural Network Applications #Computer science #Distributed computing #Distributed database #Distributed learning #Feature learning #Federated learning #Machine learning #Overhead (engineering) #Privacy-Preserving Technologies in Data #Robustness (evolution) #Stochastic Gradient Optimization Techniques #cs.AI #cs.LG

paper · pdf · doi:10.1109/tpds.2022.3157258

published in IEEE Transactions on Parallel and Distributed Systems 33(12), 3460-3477 (Institute of Electrical and Electronics Engineers)

arxiv created 2022/03/03 · openalex publication_date 2022/03/07 · arxiv updated 2022/03/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

The diversity and quantity of data warehouses, gathering data from distributed devices such as mobile devices, can enhance the success and robustness of machine learning algorithms. Federated learning enables distributed participants to collaboratively learn a commonly shared model while holding data locally. However, it is also faced with expensive communication and limitations due to the heterogeneity of distributed data sources and lack of access to global data. In this paper, we investigate a practical distributed learning scenario where multiple downstream tasks (e.g., classifiers) could be efficiently learned from dynamically updated and non-iid distributed data sources while providing local data privatization. We introduce a new distributed/collaborative learning scheme to address communication overhead via latent compression, leveraging global data while providing privatization of local data without additional cost due to encryption or perturbation. This scheme divides learning into (1) informative feature encoding, and transmitting the latent representation of local data to address communication overhead; (2) downstream tasks centralized at the server using the encoded codes gathered from each node to address computing overhead. Besides, a disentanglement strategy is applied to address the privatization of sensitive components of local data. Extensive experiments are conducted on image and speech datasets. The results demonstrate that downstream tasks with the compact latent representations with the privatization of local data can achieve comparable accuracy to centralized learning.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• WaveNet: A Generative Model for Raw Audio
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• Federated Learning: Strategies for Improving Communication Efficiency
• Practical Secure Aggregation for Federated Learning on User-Held Data
• Differentially Private Federated Learning: A Client Level Perspective
• Fast Decoding in Sequence Models using Discrete Latent Variables
• Local SGD Converges Fast and Communicates Little
• How To Backdoor Federated Learning
• Towards Federated Learning at Scale: System Design
• Neural Discrete Representation Learning
• Federated Learning in Mobile Edge Networks: A Comprehensive Survey
• Federated Learning: Challenges, Methods, and Future Directions
• Federated Learning: Opportunities and Challenges
• Split learning for health: Distributed deep learning without sharing raw patient data
• Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptron
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• Detailed comparison of communication efficiency of split learning and federated learning

Cited by

• VAE-based Feature Disentanglement for Data Augmentation and Compression in Generalized GNSS Interference Classification