2023/03/02 by Xin Gu, Gautam Kamath, Gu, Xin +3 · 1 voice · 2 citations
Computer Science · Mathematics · #Computer Vision and Pattern Recognition (cs.CV) #Cryptography and Security (cs.CR) #Data Structures and Algorithms (cs.DS) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Privacy-Preserving Technologies in Data #Statistical Methods and Bayesian Inference #Stochastic Gradient Optimization Techniques #cs.CR #cs.CV #cs.DS #cs.LG #stat.ML
paper · pdf · doi:10.48550/arxiv.2303.01256
openalex publication_date 2023/03/02 · arxiv published 2023/03/02 · arxiv updated 2025/07/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Differentially private stochastic gradient descent privatizes model training by injecting noise into each iteration, where the noise magnitude increases with the number of model parameters. Recent works suggest that we can reduce the noise by leveraging public data for private machine learning, by projecting gradients onto a subspace prescribed by the public data. However, given a choice of public datasets, it is not a priori clear which one may be most appropriate for the private task. We give an algorithm for selecting a public dataset by measuring a low-dimensional subspace distance between gradients of the public and private examples. We provide theoretical analysis demonstrating that the excess risk scales with this subspace distance. This distance is easy to compute and robust to modifications in the setting. Empirical evaluation shows that trained model accuracy is monotone in this distance.