Differentially Private Federated Learning on Heterogeneous Data

2021/11/17 by Maxence Noble, Noble, Maxence, Aurélien Bellet +3 · 6 citations
Computer Science · #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Mobile Crowdsensing and Crowdsourcing #Privacy-Preserving Technologies in Data #Statistics Theory (math.ST) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2111.09278

openalex publication_date 2021/11/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Federated Learning (FL) is a paradigm for large-scale distributed learning which faces two key challenges: (i) efficient training from highly heterogeneous user data, and (ii) protecting the privacy of participating users. In this work, we propose a novel FL approach (DP-SCAFFOLD) to tackle these two challenges together by incorporating Differential Privacy (DP) constraints into the popular SCAFFOLD algorithm. We focus on the challenging setting where users communicate with a "honest-but-curious" server without any trusted intermediary, which requires to ensure privacy not only towards a third-party with access to the final model but also towards the server who observes all user communications. Using advanced results from DP theory, we establish the convergence of our algorithm for convex and non-convex objectives. Our analysis clearly highlights the privacy-utility trade-off under data heterogeneity, and demonstrates the superiority of DP-SCAFFOLD over the state-of-the-art algorithm DP-FedAvg when the number of local updates and the level of heterogeneity grow. Our numerical results confirm our analysis and show that DP-SCAFFOLD provides significant gains in practice.

Citations

• Adam: A Method for Stochastic Optimization
• Local Privacy, Data Processing Inequalities, and Statistical Minimax Rates
• Differentially Private Learning with Adaptive Clipping
• Federated Learning With Differential Privacy: Algorithms and Performance Analysis
• Shuffled Model of Federated Learning: Privacy, Communication and\n Accuracy Trade-offs
• Federated Learning Based on Dynamic Regularization

Cited by

• First Provable Guarantees for Practical Private FL: Beyond Restrictive Assumptions
• pFedBBN: A Personalized Federated Test-Time Adaptation with Balanced Batch Normalization for Class-Imbalanced Data
• Differential Privacy for Euclidean Jordan Algebra with Applications to Private Symmetric Cone Programming
• FedMentor: Domain-Aware Differential Privacy for Heterogeneous Federated LLMs in Mental Health
• Securing Private Federated Learning in a Malicious Setting: A Scalable TEE-Based Approach with Client Auditing
• From Noisy Fixed-Point Iterations to Private ADMM for Centralized and Federated Learning

Related

• Federated Learning with Heterogeneous Differential Privacy
• Accelerating Differentially Private Federated Learning via Adaptive Extrapolation
• Federated Learning with Differential Privacy: An Utility-Enhanced Approach
• Differential Privacy Meets Federated Learning under Communication Constraints
• Federated Learning with Differential Privacy