Statistical Inference for Differentially Private Stochastic Gradient Descent

2025/07/28 by Xia, Xintao, Zhang, Linjun, Cai, Zhanrui
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Methodology (stat.ME)

paper · doi:10.48550/arxiv.2507.20560

Abstract

Privacy preservation in machine learning, particularly through Differentially Private Stochastic Gradient Descent (DP-SGD), is critical for sensitive data analysis. However, existing statistical inference methods for SGD predominantly focus on cyclic subsampling, while DP-SGD requires randomized subsampling. This paper first bridges this gap by establishing the asymptotic properties of SGD under the randomized rule and extending these results to DP-SGD. For the output of DP-SGD, we show that the asymptotic variance decomposes into statistical, sampling, and privacy-induced components. Two methods are proposed for constructing valid confidence intervals: the plug-in method and the random scaling method. We also perform extensive numerical analysis, which shows that the proposed confidence intervals achieve nominal coverage rates while maintaining privacy.

Citations

• Revisiting Gradient Clipping: Stochastic bias and tight convergence guarantees
• Online Statistical Inference for Contextual Bandits via Stochastic Gradient Descent
• Privacy of Noisy Stochastic Gradient Descent: More Iterations without More Privacy Loss
• Opacus: User-Friendly Differential Privacy Library in PyTorch
• Deep Reinforcement Learning at the Edge of the Statistical Precipice
• Differentially private inference via noisy optimization
• Differential Privacy Dynamics of Langevin Diffusion and Noisy Gradient Descent
• Understanding Gradient Clipping in Private SGD: A Geometric Perspective
• Stability of Stochastic Gradient Descent on Nonsmooth Convex Losses
• Deep Learning with Gaussian Differential Privacy
• Why gradient clipping accelerates training: A theoretical justification for adaptivity
• The Cost of Privacy: Optimal Rates of Convergence for Parameter Estimation with Differential Privacy
• Privacy Amplification by Subsampling: Tight Analyses via Couplings and Divergences
• Uncertainty Quantification for Online Learning and Stochastic Approximation via Hierarchical Incremental Gradient Descent
• Attention Is All You Need
• Statistical Inference for Model Parameters in Stochastic Gradient Descent
• Differentially Private Ordinary Least Squares
• Acceleration of Stochastic Approximation by Averaging

Related

• Individual Privacy Accounting for Differentially Private Stochastic Gradient Descent
• An Improved Privacy and Utility Analysis of Differentially Private SGD with Bounded Domain and Smooth Losses
• Decentralized Differentially Private Without-Replacement Stochastic Gradient Descent
• Differentially Private Stochastic Gradient Descent with Low-Noise
• Differentially Private Learning Needs Hidden State (Or Much Faster Convergence)