Robust Machine Learning via Privacy/Rate-Distortion Theory

2020/07/22 by Ye Wang, Wang, Ye, Shuchin Aeron +7
Computer Science · Social Sciences · #Adversarial Robustness in Machine Learning #Computer Science and Game Theory (cs.GT) #Cryptography and Security (cs.CR) #Ethics and Social Impacts of AI #FOS: Computer and information sciences #Information Theory (cs.IT) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Privacy-Preserving Technologies in Data

paper · pdf · doi:10.48550/arxiv.2007.11693

openalex publication_date 2020/07/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Robust machine learning formulations have emerged to address the prevalent vulnerability of deep neural networks to adversarial examples. Our work draws the connection between optimal robust learning and the privacy-utility tradeoff problem, which is a generalization of the rate-distortion problem. The saddle point of the game between a robust classifier and an adversarial perturbation can be found via the solution of a maximum conditional entropy problem. This information-theoretic perspective sheds light on the fundamental tradeoff between robustness and clean data performance, which ultimately arises from the geometric structure of the underlying data distribution and perturbation constraints.

Citations

• Distributionally Robust Stochastic Optimization with Wasserstein Distance
• Certifying Some Distributional Robustness with Principled Adversarial\n Training
• Machine vs Machine: Minimax-Optimal Defense Against Adversarial Examples
• Optimal Transport Based Distributionally Robust Optimization: Structural Properties and Iterative Schemes

Related

• Robustness, Privacy, and Generalization of Adversarial Training
• Bridging Privacy and Robustness for Trustworthy Machine Learning
• From Robustness to Privacy and Back
• Balancing Privacy, Robustness, and Efficiency in Machine Learning
• Robustness Threats of Differential Privacy