Wide Network Learning with Differential Privacy

2021/03/01 by Huanyu Zhang, Zhang, Huanyu, Ilya Mironov +3 · 9 citations
Computer Science · #Adversarial Robustness in Machine Learning #Artificial intelligence #Artificial neural network #Computer science #Contrast (vision) #Curse of dimensionality #Data Structures and Algorithms (cs.DS) #Data mining #Deep learning #Deep neural networks #Differential privacy #Embedding #Empirical risk minimization #FOS: Computer and information sciences #Focus (optics) #Intuition #Machine Learning (cs.LG) #Machine learning #Minification #Privacy-Preserving Technologies in Data #Stochastic Gradient Optimization Techniques #cs.DS #cs.LG

paper · pdf · doi:10.48550/arxiv.2103.01294

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2021/03/01 · arxiv created 2021/06/04 · arxiv updated 2021/06/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Despite intense interest and considerable effort, the current generation of neural networks suffers a significant loss of accuracy under most practically relevant privacy training regimes. One particularly challenging class of neural networks are the wide ones, such as those deployed for NLP typeahead prediction or recommender systems. Observing that these models share something in common--an embedding layer that reduces the dimensionality of the input--we focus on developing a general approach towards training these models that takes advantage of the sparsity of the gradients. More abstractly, we address the problem of differentially private empirical risk minimization (ERM) for models that admit sparse gradients. We demonstrate that for non-convex ERM problems, the loss is logarithmically dependent on the number of parameters, in contrast with polynomial dependence for the general case. Following the same intuition, we propose a novel algorithm for privately training neural networks. Finally, we provide an empirical study of a DP wide neural network on a real-world dataset, which has been rarely explored in the previous work.

Citations

• Calibrating Noise to Sensitivity in Private Data Analysis
• Distributed Representations of Words and Phrases and their Compositionality
• Learning Differentially Private Recurrent Language Models
• Differentially Private Data Generative Models
• The Cost of Privacy: Optimal Rates of Convergence for Parameter Estimation with Differential Privacy
• Differentially Private Learning with Adaptive Clipping
• Subsampled Rényi Differential Privacy and Analytical Moments Accountant
• Differentially Private Empirical Risk Minimization Revisited: Faster and More General
• Rényi Differential Privacy of the Sampled Gaussian Mechanism
• Training Production Language Models without Memorizing User Data
• Differentially Private Deep Learning with Direct Feedback Alignment
• Efficient Privacy-Preserving Stochastic Nonconvex Optimization
• Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identification

Cited by

• Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)

Related

• Fine-Tuning with Differential Privacy Necessitates an Additional Hyperparameter Search
• Deep Learning with Differential Privacy
• An Optimization Framework for Differentially Private Sparse Fine-Tuning
• Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learning
• Differentially Private Deep Learning with ModelMix