2021/06/13 by Rudrajit Das, Abolfazl Hashemi, Das, Rudrajit +5 · 1 citation
Computer Science · #Cryptography and Data Security #Distributed #FOS: Computer and information sciences #FOS: Electrical engineering #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Optimization and Control (math.OC) #Parallel #Privacy-Preserving Technologies in Data #Signal Processing (eess.SP) #Stochastic Gradient Optimization Techniques #and Cluster Computing (cs.DC) #electronic engineering #information engineering
paper · pdf · doi:10.48550/arxiv.2106.07094
openalex publication_date 2021/06/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
There is a dearth of convergence results for differentially private federated learning (FL) with non-Lipschitz objective functions (i.e., when gradient norms are not bounded). The primary reason for this is that the clipping operation (i.e., projection onto an ℓ2 ball of a fixed radius called the clipping threshold) for bounding the sensitivity of the average update to each client's update introduces bias depending on the clipping threshold and the number of local steps in FL, and analyzing this is not easy. For Lipschitz functions, the Lipschitz constant serves as a trivial clipping threshold with zero bias. However, Lipschitzness does not hold in many practical settings; moreover, verifying it and computing the Lipschitz constant is hard. Thus, the choice of the clipping threshold is non-trivial and requires a lot of tuning in practice. In this paper, we provide the first convergence result for private FL on smooth convex objectives for a general clipping threshold -- without assuming Lipschitzness. We also look at a simpler alternative to clipping (for bounding sensitivity) which is normalization -- where we use only a scaled version of the unit vector along the client updates, completely discarding the magnitude information. The resulting normalization-based private FL algorithm is theoretically shown to have better convergence than its clipping-based counterpart on smooth convex functions. We corroborate our theory with synthetic experiments as well as experiments on benchmarking datasets.