Salvaging Federated Learning by Local Adaptation

2020/02/12 by Changyuan Yu, Yu, Tao, Eugene Bagdasaryan +3 · 7 citations
Computer Science · Social Sciences · #Artificial Intelligence (cs.AI) #Distributed #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Mobile Crowdsensing and Crowdsourcing #Parallel #Privacy, Security, and Data Protection #Privacy-Preserving Technologies in Data #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2002.04758

openalex publication_date 2020/02/12 · openalex created_date 2020/02/24 · openalex updated_date 2026/07/28

Abstract

Federated learning (FL) is a heavily promoted approach for training ML models on sensitive data, e.g., text typed by users on their smartphones. FL is expressly designed for training on data that are unbalanced and non-iid across the participants. To ensure privacy and integrity of the fedeated model, latest FL approaches use differential privacy or robust aggregation. We look at FL from the local viewpoint of an individual participant and ask: (1) do participants have an incentive to participate in FL? (2) how can participants individually improve the quality of their local models, without re-designing the FL framework and/or involving other participants? First, we show that on standard tasks such as next-word prediction, many participants gain no benefit from FL because the federated model is less accurate on their data than the models they can train locally on their own. Second, we show that differential privacy and robust aggregation make this problem worse by further destroying the accuracy of the federated model for many participants. Then, we evaluate three techniques for local adaptation of federated models: fine-tuning, multi-task learning, and knowledge distillation. We analyze where each is applicable and demonstrate that all participants benefit from local adaptation. Participants whose local models are poor obtain big accuracy improvements over conventional FL. Participants whose local models are better than the federated model\textemdash and who have no incentive to participate in FL today\textemdash improve less, but sufficiently to make the adapted federated model better than their local models.

Citations

• Distilling the Knowledge in a Neural Network
• Learning Differentially Private Recurrent Language Models
• How To Backdoor Federated Learning
• One-Shot Federated Learning
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• Advances and Open Problems in Federated Learning
• Personalized Federated Learning: A Meta-Learning Approach
• Personalized Federated Learning with Moreau Envelopes
• Learning Multiple Layers of Features from Tiny Images

Cited by

• A Closer Look at Personalized Fine-Tuning in Heterogeneous Federated Learning
• Adaptive collaboration for online personalized distributed learning with heterogeneous clients
• Spatio-Temporal Representation Decoupling and Enhancement for Federated Instrument Segmentation in Surgical Videos
• FeDa4Fair: Client-Level Federated Datasets for Fairness Evaluation
• Black-Box Privacy Attacks on Shared Representations in Multitask Learning
• Differentially Private Learning Needs Better Features (or Much More Data)
• Optimality and Stability in Federated Learning: A Game-theoretic Approach

Related

• Enhancing the Performance of Global Model by Improving the Adaptability of Local Models in Federated Learning
• Federated Reconstruction: Partially Local Federated Learning
• FedBR: Improving Federated Learning on Heterogeneous Data via Local Learning Bias Reduction
• Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning
• Private Federated Learning with Domain Adaptation