FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusion

2021/10/21 by Sijie Cheng, Cheng, Sijie, Jingwen Wu +5 · 10 citations
Computer Science · Medicine · #Artificial Intelligence (cs.AI) #COVID-19 diagnosis using AI #Cryptography and Data Security #FOS: Computer and information sciences #Machine Learning (cs.LG) #Privacy-Preserving Technologies in Data #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.2110.11027

Under review as a conference paper at ICLR 2022

openalex publication_date 2021/10/21 · arxiv created 2021/12/07 · arxiv updated 2021/12/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Today data is often scattered among billions of resource-constrained edge devices with security and privacy constraints. Federated Learning (FL) has emerged as a viable solution to learn a global model while keeping data private, but the model complexity of FL is impeded by the computation resources of edge nodes. In this work, we investigate a novel paradigm to take advantage of a powerful server model to break through model capacity in FL. By selectively learning from multiple teacher clients and itself, a server model develops in-depth knowledge and transfers its knowledge back to clients in return to boost their respective performance. Our proposed framework achieves superior performance on both server and client models and provides several advantages in a unified framework, including flexibility for heterogeneous client architectures, robustness to poisoning attacks, and communication efficiency between clients and server on various image classification tasks.

Citations

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Federated Multi-Task Learning
• NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications
• Don't Use Large Mini-Batches, Use Local SGD
• LEAF: A Benchmark for Federated Settings
• Towards Federated Learning at Scale: System Design
• Fair Resource Allocation in Federated Learning
• FedMD: Heterogenous Federated Learning via Model Distillation
• Abnormal Client Behavior Detection in Federated Learning
• Advances and Open Problems in Federated Learning
• Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer
• Improved protein structure prediction using potentials from deep learning
• Language Models are Few-Shot Learners
• Ensemble Distillation for Robust Model Fusion in Federated Learning
• FedML: A Research Library and Benchmark for Federated Machine Learning
• Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge
• Reinforced Multi-Teacher Selection for Knowledge Distillation

Cited by

• Rethinking Knowledge Distillation in Collaborative Machine Learning: Memory, Knowledge, and Their Interactions
• FedeCouple: Fine-Grained Balancing of Global-Generalization and Local-Adaptability in Federated Learning
• FedMMKT:Co-Enhancing a Server Text-to-Image Model and Client Task Models in Multi-Modal Federated Learning
• Choice Outweighs Effort: Facilitating Complementary Knowledge Fusion in Federated Learning via Re-calibration and Merit-discrimination
• Towards a Larger Model via One-Shot Federated Learning on Heterogeneous Client Models
• Boosting Generalization Performance in Model-Heterogeneous Federated Learning Using Variational Transposed Convolution
• Federated Foundation Models Fine-Tuning with Heterogeneous Compressed Clients
• Avoid Forgetting by Preserving Global Knowledge Gradients in Federated Learning with Non-IID Data
• Privacy-Preserving Federated Embedding Learning for Localized Retrieval-Augmented Generation
• Towards Harnessing the Collaborative Power of Large and Small Models for Domain Tasks

Related

• Enhancing Heterogeneous Federated Learning with Knowledge Extraction and Multi-Model Fusion
• The More is not the Merrier: Investigating the Effect of Client Size on Federated Learning
• Federated Generative Learning with Foundation Models
• A Survey on Model-heterogeneous Federated Learning: Problems, Methods, and Prospects
• GeFL: Model-Agnostic Federated Learning with Generative Models