FedGKD: Toward Heterogeneous Federated Learning via Global Knowledge Distillation

2023/09/15 by Dezhong Yao, Wanning Pan, Yutong Dai +7 · 87 citations
Computer Science · #Artificial intelligence #Computer science #Convergence (economics) #Internet Traffic Analysis and Secure E-voting #Machine learning #Mobile Crowdsensing and Crowdsourcing #Privacy-Preserving Technologies in Data

paper · doi:10.1109/tc.2023.3315066

published in IEEE Transactions on Computers 73(1), 3-17 (Institute of Electrical and Electronics Engineers)

openalex publication_date 2023/09/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/31

Abstract

Federated learning, as one enabling technology of edge intelligence, has gained substantial attention due to its efficacy in training deep learning models without data privacy and network bandwidth concerns. However, due to the heterogeneity of the edge computing system and data, many methods suffer from the“client-drift”issue that could considerably impede the convergence of global model training: local models on clients can drift apart, and the aggregated model can be different from the global optimum. To tackle this issue, one intuitive idea is to guide the local model training by global teachers,i.e.,past global models, where each client learns the global knowledge from past global models via adaptive knowledge distillation techniques. Inspired by these insights, we propose a novel approach for heterogeneous federated learning,FedGKD, which fuses the knowledge from historical global models and guides local training to alleviate the“client-drift”issue. In this paper, we evaluateFedGKDthrough extensive experiments across various CV and NLP datasets (i.e.,CIFAR-10/100, Tiny-ImageNet, AG News, SST5) under different heterogeneous settings. The proposed method is guaranteed to converge under common assumptions and outperforms the state-of-the-art baselines in the non-IID federated setting.

Cited by

• FedMMKT:Co-Enhancing a Server Text-to-Image Model and Client Task Models in Multi-Modal Federated Learning
• FedPromo: Federated Lightweight Proxy Models at the Edge Bring New Domains to Foundation Models
• A Novel Algorithm for Personalized Federated Learning: Knowledge Distillation with Weighted Combination Loss
• FedNolowe: A normalized loss-based weighted aggregation strategy for robust federated learning in heterogeneous environments. [europepmc]
• Achieving consistency in FedSAM using local adaptive distillation on sports image classification. [europepmc]
• Hallucination-aware learning and latency optimization transformer (HALL-OPT) for real-time edge intelligence. [europepmc]
• A Review of Federated Large Language Models for Industry 4.0 [europepmc]

Related

• Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Data
• FedSKC: Federated Learning with Non-IID Data via Structural Knowledge Collaboration
• FedIN: Federated Intermediate Layers Learning for Model Heterogeneity
• Closing the Gap between Client and Global Model Performance in Heterogeneous Federated Learning
• FedSiKD: Clients Similarity and Knowledge Distillation: Addressing Non-i.i.d. and Constraints in Federated Learning