Efficient and Private Federated Learning with Partially Trainable Networks

2021/10/06 by Hakim Sidahmed, Sidahmed, Hakim, Zheng Xu +7 · 3 citations
Computer Science · #Age of Information Optimization #FOS: Computer and information sciences #Machine Learning (cs.LG) #Privacy-Preserving Technologies in Data #Stochastic Gradient Optimization Techniques #cs.LG

paper · pdf · doi:10.48550/arxiv.2110.03450

V2: minor wording and format improvement

openalex publication_date 2021/10/06 · openalex created_date 2021/10/11 · arxiv created 2021/11/08 · arxiv updated 2021/11/09 · openalex updated_date 2026/07/28

Abstract

Federated learning is used for decentralized training of machine learning models on a large number (millions) of edge mobile devices. It is challenging because mobile devices often have limited communication bandwidth and local computation resources. Therefore, improving the efficiency of federated learning is critical for scalability and usability. In this paper, we propose to leverage partially trainable neural networks, which freeze a portion of the model parameters during the entire training process, to reduce the communication cost with little implications on model performance. Through extensive experiments, we empirically show that Federated learning of Partially Trainable neural networks (FedPT) can result in superior communication-accuracy trade-offs, with up to 46× reduction in communication cost, at a small accuracy cost. Our approach also enables faster training, with a smaller memory footprint, and better utility for strong differential privacy guarantees. The proposed FedPT method can be particularly interesting for pushing the limitations of over-parameterization in on-device learning.

Citations

• Distilling the Knowledge in a Neural Network
• Do Deep Nets Really Need to be Deep?
• EMNIST: an extension of MNIST to handwritten letters
• LEAF: A Benchmark for Federated Settings
• Towards Federated Learning at Scale: System Design
• Echo State Neural Machine Translation
• Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs
• Training Production Language Models without Memorizing User Data
• Rethinking Attention with Performers
• Learning Multiple Layers of Features from Tiny Images
• Federated Reconstruction: Partially Local Federated Learning
• A Field Guide to Federated Optimization
• Advances and Open Problems in Federated Learning

Cited by

• Energy and Memory-Efficient Federated Learning With Ordered Layer Freezing
• Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)
• FedPromo: Federated Lightweight Proxy Models at the Edge Bring New Domains to Foundation Models

Related

• Partial Variable Training for Efficient On-Device Federated Learning
• Straggler-resilient Federated Learning: Tackling Computation Heterogeneity with Layer-wise Partial Model Training in Mobile Edge Network
• Energy and Memory-Efficient Federated Learning With Ordered Layer Freezing
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Evaluating the Communication Efficiency in Federated Learning Algorithms