Parameterized Knowledge Transfer for Personalized Federated Learning

2021/11/04 by Jie Zhang, Zhang, Jie, Song Guo +9 · 12 citations
Computer Science · #Distributed #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Parallel #Privacy-Preserving Technologies in Data #Recommender Systems and Techniques #and Cluster Computing (cs.DC) #cs.DC #cs.LG

paper · pdf · doi:10.48550/arxiv.2111.02862

arxiv created 2021/11/04 · openalex publication_date 2021/11/04 · arxiv updated 2021/11/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In recent years, personalized federated learning (pFL) has attracted increasing attention for its potential in dealing with statistical heterogeneity among clients. However, the state-of-the-art pFL methods rely on model parameters aggregation at the server side, which require all models to have the same structure and size, and thus limits the application for more heterogeneous scenarios. To deal with such model constraints, we exploit the potentials of heterogeneous model settings and propose a novel training framework to employ personalized models for different clients. Specifically, we formulate the aggregation procedure in original pFL into a personalized group knowledge transfer training algorithm, namely, KT-pFL, which enables each client to maintain a personalized soft prediction at the server side to guide the others' local training. KT-pFL updates the personalized soft prediction of each client by a linear combination of all local soft predictions using a knowledge coefficient matrix, which can adaptively reinforce the collaboration among clients who own similar data distribution. Furthermore, to quantify the contributions of each client to others' personalized training, the knowledge coefficient matrix is parameterized so that it can be trained simultaneously with the models. The knowledge coefficient matrix and the model parameters are alternatively updated in each round following the gradient descent way. Extensive experiments on various datasets (EMNIST, Fashion_MNIST, CIFAR-10) are conducted under different settings (heterogeneous models and data distributions). It is demonstrated that the proposed framework is the first federated learning paradigm that realizes personalized model training via parameterized group knowledge transfer while achieving significant performance gain comparing with state-of-the-art algorithms.

Citations

• Gradient-based learning applied to document recognition
• ImageNet classification with deep convolutional neural networks
• One-Shot Federated Learning
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• FedMD: Heterogenous Federated Learning via Model Distillation
• Federated Learning with Personalization Layers
• Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer
• Three Approaches for Personalization with Applications to Federated Learning
• Learning Multiple Layers of Features from Tiny Images
• Federated Mixture of Experts

Cited by

• SAM-Fed: SAM-Guided Federated Semi-Supervised Learning for Medical Image Segmentation
• CO-PFL: Contribution-Oriented Personalized Federated Learning for Heterogeneous Networks
• Beyond Static Knowledge Messengers: Towards Adaptive, Fair, and Scalable Federated Learning for Medical AI
• Towards Privacy-Preserving and Heterogeneity-aware Split Federated Learning via Probabilistic Masking
• Personalized Subgraph Federated Learning with Sheaf Collaboration
• Hypernetworks for Model-Heterogeneous Personalized Federated Learning
• FedVLM: Scalable Personalized Vision-Language Models through Federated Learning
• Addressing the Collaboration Dilemma in Low-Data Federated Learning via Transient Sparsity
• Soft-Label Caching and Sharpening for Communication-Efficient Federated Distillation
• Towards Harnessing the Collaborative Power of Large and Small Models for Domain Tasks
• FedCARE: A Multi-Objective Personalised Federated Learning Framework for Smart Healthcare
• Federated Hierarchical Reinforcement Learning for Adaptive Traffic Signal Control

Related

• Learn What You Need in Personalized Federated Learning
• Personalized Federated Learning for Heterogeneous Clients with Clustered Knowledge Transfer
• Personalizing or Not: Dynamically Personalized Federated Learning with Incentives
• Personalized Federated Learning with Adaptive Feature Aggregation and Knowledge Transfer
• Decoupling General and Personalized Knowledge in Federated Learning via Additive and Low-Rank Decomposition