Federated Model Distillation with Noise-Free Differential Privacy

2020/09/11 by Lichao Sun, Lingjuan Lyu, Sun, Lichao +1 · 7 citations
Computer Science · Engineering · Mathematics · #Cryptography and Data Security #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Privacy-Preserving Technologies in Data #Vehicular Ad Hoc Networks (VANETs) #cs.CR #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2009.05537

accepted by IJCAI-21

openalex publication_date 2020/09/11 · arxiv created 2021/05/21 · arxiv updated 2021/05/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Conventional federated learning directly averages model weights, which is only possible for collaboration between models with homogeneous architectures. Sharing prediction instead of weight removes this obstacle and eliminates the risk of white-box inference attacks in conventional federated learning. However, the predictions from local models are sensitive and would leak training data privacy to the public. To address this issue, one naive approach is adding the differentially private random noise to the predictions, which however brings a substantial trade-off between privacy budget and model performance. In this paper, we propose a novel framework called FEDMD-NFDP, which applies a Noise-Free Differential Privacy (NFDP) mechanism into a federated model distillation framework. Our extensive experimental results on various datasets validate that FEDMD-NFDP can deliver not only comparable utility and communication efficiency but also provide a noise-free differential privacy guarantee. We also demonstrate the feasibility of our FEDMD-NFDP by considering both IID and non-IID setting, heterogeneous model architectures, and unlabelled public datasets from a different distribution.

Citations

• Distilling the Knowledge in a Neural Network
• Prototypical Networks for Few-shot Learning
• Differentially Private Federated Learning: A Client Level Perspective
• LEAF: A Benchmark for Federated Settings
• FedMD: Heterogenous Federated Learning via Model Distillation
• Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer
• LDP-FL: Practical Private Aggregation in Federated Learning with Local Differential Privacy

Cited by

• Rethinking Knowledge Distillation in Collaborative Machine Learning: Memory, Knowledge, and Their Interactions
• On the Convergence and Stability of Distributed Sub-model Training
• Boosting Generalization Performance in Model-Heterogeneous Federated Learning Using Variational Transposed Convolution
• Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge
• Fair and Differentially Private Distributed Frequency Estimation
• Differentially Private Representation for NLP: Formal Guarantee and An Empirical Study on Privacy and Fairness
• Source Inference Attacks in Federated Learning
• A Reputation Mechanism Is All You Need: Collaborative Fairness and Adversarial Robustness in Federated Learning

Related

• Federated Learning with Enhanced Privacy via Model Splitting and Random Client Participation
• Federated Learning with Sparsified Model Perturbation: Improving Accuracy under Client-Level Differential Privacy
• Federated Learning with Heterogeneous Differential Privacy
• Universally Harmonizing Differential Privacy Mechanisms for Federated Learning: Boosting Accuracy and Convergence
• Noise-Aware Algorithm for Heterogeneous Differentially Private Federated Learning