Maximum-Entropy Adversarial Data Augmentation for Improved\n Generalization and Robustness

2020/10/15 by L. Zhao, Ting Liu, Zhao, Long +5 · 2 citations
Computer Science · Engineering · #Advanced Image Processing Techniques #Adversarial Robustness in Machine Learning #Anomaly Detection Techniques and Applications #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Image Processing Techniques and Applications #Image and Signal Denoising Methods #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2010.08001

openalex publication_date 2020/10/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Adversarial data augmentation has shown promise for training robust deep\nneural networks against unforeseen data shifts or corruptions. However, it is\ndifficult to define heuristics to generate effective fictitious target\ndistributions containing "hard" adversarial perturbations that are largely\ndifferent from the source distribution. In this paper, we propose a novel and\neffective regularization term for adversarial data augmentation. We\ntheoretically derive it from the information bottleneck principle, which\nresults in a maximum-entropy formulation. Intuitively, this regularization term\nencourages perturbing the underlying source distribution to enlarge predictive\nuncertainty of the current model, so that the generated "hard" adversarial\nperturbations can improve the model robustness during training. Experimental\nresults on three standard benchmarks demonstrate that our method consistently\noutperforms the existing state of the art by a statistically significant\nmargin.\n

Citations

• Auto-Encoding Variational Bayes
• On Information and Sufficiency
• Gradient-based learning applied to document recognition
• Estimation of Entropy and Mutual Information
• Backpropagation Applied to Handwritten Zip Code Recognition
• ImageNet classification with deep convolutional neural networks
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Humanoid-inspired Causal Representation Learning for Domain Generalization
• Target-Oriented Single Domain Generalization

Related

• Rethinking Uncertainty in Deep Learning: Whether and How it Improves Robustness
• Improving Adversarial Robustness by Putting More Regularizations on Less Robust Samples
• Revisiting Hilbert-Schmidt Information Bottleneck for Adversarial Robustness
• Boosting Adversarial Training via Fisher-Rao Norm-based Regularization
• Boosting Adversarial Robustness From The Perspective of Effective Margin Regularization