Personalized Federated Learning for Heterogeneous Clients with Clustered Knowledge Transfer

2021/09/16 by Yae Jee Cho, Jianyu Wang, Cho, Yae Jee +5 · 4 citations
Computer Science · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Privacy-Preserving Technologies in Data #Stochastic Gradient Optimization Techniques #cs.LG

paper · pdf · doi:10.48550/arxiv.2109.08119

arxiv created 2021/09/16 · openalex publication_date 2021/09/16 · arxiv updated 2021/09/17 · openalex created_date 2021/09/27 · openalex updated_date 2026/07/28

Abstract

Personalized federated learning (FL) aims to train model(s) that can perform well for individual clients that are highly data and system heterogeneous. Most work in personalized FL, however, assumes using the same model architecture at all clients and increases the communication cost by sending/receiving models. This may not be feasible for realistic scenarios of FL. In practice, clients have highly heterogeneous system-capabilities and limited communication resources. In our work, we propose a personalized FL framework, PerFed-CKT, where clients can use heterogeneous model architectures and do not directly communicate their model parameters. PerFed-CKT uses clustered co-distillation, where clients use logits to transfer their knowledge to other clients that have similar data-distributions. We theoretically show the convergence and generalization properties of PerFed-CKT and empirically show that PerFed-CKT achieves high test accuracy with several orders of magnitude lower communication cost compared to the state-of-the-art personalized FL schemes.

Citations

• Towards Federated Learning at Scale: System Design
• Large scale distributed neural network training through online distillation
• Learning Multiple Layers of Features from Tiny Images
• Advances and Open Problems in Federated Learning

Cited by

• Rethinking Knowledge Distillation in Collaborative Machine Learning: Memory, Knowledge, and Their Interactions
• Towards Personalized Quantum Federated Learning for Anomaly Detection
• Towards a Larger Model via One-Shot Federated Learning on Heterogeneous Client Models
• Whom to Trust? Adaptive Collaboration in Personalized Federated Learning

Related

• Parameterized Knowledge Transfer for Personalized Federated Learning
• Personalizing or Not: Dynamically Personalized Federated Learning with Incentives
• Client-supervised Federated Learning: Towards One-model-for-all Personalization
• On the Convergence of Clustered Federated Learning
• Personalized Federated Learning with First Order Model Optimization