Generative Models for Effective ML on Private, Decentralized Datasets

2019/11/15 by Sean Augenstein, H. Brendan McMahan, Augenstein, Sean +14 · 43 citations
Computer Science · Mathematics · #Adversarial Robustness in Machine Learning #Artificial intelligence #Computer science #Data mining #Debugging #Differential privacy #Enhanced Data Rates for GSM Evolution #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Federated learning #Generative grammar #Generative model #Intuition #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine learning #Outlier #Privacy-Preserving Technologies in Data #Raw data #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1911.06679

published in arXiv (Cornell University) (Cornell University) · 26 pages, 8 figures. Camera-ready ICLR 2020 version

openalex publication_date 2019/11/15 · arxiv created 2020/02/04 · arxiv updated 2020/02/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

To improve real-world applications of machine learning, experienced modelers develop intuition about their datasets, their models, and how the two interact. Manual inspection of raw data - of representative samples, of outliers, of misclassifications - is an essential tool in a) identifying and fixing problems in the data, b) generating new modeling hypotheses, and c) assigning or refining human-provided labels. However, manual data inspection is problematic for privacy sensitive datasets, such as those representing the behavior of real-world individuals. Furthermore, manual data inspection is impossible in the increasingly important setting of federated learning, where raw examples are stored at the edge and the modeler may only access aggregated outputs such as metrics or model parameters. This paper demonstrates that generative models - trained using federated methods and with formal differential privacy guarantees - can be used effectively to debug many commonly occurring data issues even when the data cannot be directly inspected. We explore these methods in applications to text with differentially private federated RNNs and to images using a novel algorithm for differentially private federated GANs.

Citations

• Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition
• Calibrating Noise to Sensitivity in Private Data Analysis
• Learning Differentially Private Recurrent Language Models
• Differentially Private Generative Adversarial Network
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• LEAF: A Benchmark for Federated Settings
• Applied Federated Learning: Improving Google Keyboard Query Suggestions
• Reconstruction and Membership Inference Attacks against Generative Models

Cited by

• Toward provably private analytics and insights into GenAI use
• Federated Flow Matching
• Differentially Private Decentralized Dataset Synthesis Through Randomized Mixing with Correlated Noise
• Decentralized Federated Learning of Probabilistic Generative Classifiers
• InvisibleInk: High-Utility and Low-Cost Text Generation with Differential Privacy
• Robust Federated Learning with Confidence-Weighted Filtering and GAN-Based Completion under Noisy and Incomplete Data

Related

• Federated Generative Privacy
• Differentially Private Synthetic Data: Applied Evaluations and Enhancements
• Generating Higher-Fidelity Synthetic Datasets with Privacy Guarantees
• Differentially Private Mixture of Generative Neural Networks
• Generative Models with Information-Theoretic Protection Against Membership Inference Attacks