Communication-Efficient Federated Distillation

2020/12/01 by Felix Sattler, Arturo Marbán, Sattler, Felix +5 · 3 citations
Computer Science · #Advanced Graph Neural Networks #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Privacy-Preserving Technologies in Data #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2012.00632

openalex publication_date 2020/12/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Communication constraints are one of the major challenges preventing the wide-spread adoption of Federated Learning systems. Recently, Federated Distillation (FD), a new algorithmic paradigm for Federated Learning with fundamentally different communication properties, emerged. FD methods leverage ensemble distillation techniques and exchange model outputs, presented as soft labels on an unlabeled public data set, between the central server and the participating clients. While for conventional Federated Learning algorithms, like Federated Averaging (FA), communication scales with the size of the jointly trained model, in FD communication scales with the distillation data set size, resulting in advantageous communication properties, especially when large models are trained. In this work, we investigate FD from the perspective of communication efficiency by analyzing the effects of active distillation-data curation, soft-label quantization and delta-coding techniques. Based on the insights gathered from this analysis, we present Compressed Federated Distillation (CFD), an efficient Federated Distillation method. Extensive experiments on Federated image classification and language modeling problems demonstrate that our method can reduce the amount of communication necessary to achieve fixed performance targets by more than two orders of magnitude, when compared to FD and by more than four orders of magnitude when compared with FA.

Citations

• ImageNet classification with deep convolutional neural networks
• Ternary Weight Networks
• Pointer Sentinel Mixture Models
• One-Shot Federated Learning
• LLM-Supported Manufacturing Mapping Generation
• FedMD: Heterogenous Federated Learning via Model Distillation
• Advances and Open Problems in Federated Learning
• Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer
• FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning

Cited by

• Rethinking Knowledge Distillation in Collaborative Machine Learning: Memory, Knowledge, and Their Interactions
• Federated Distillation on Edge Devices: Efficient Client-Side Filtering for Non-IID Data
• Hypernetworks for Model-Heterogeneous Personalized Federated Learning

Related

• Soft-Label Caching and Sharpening for Communication-Efficient Federated Distillation
• Federated Distillation: A Survey
• Improving Communication Efficiency of Federated Distillation via Accumulating Local Updates
• Communication-Efficient Federated Distillation with Active Data Sampling
• Federated Learning with Label-Masking Distillation