A Flat Minima Perspective on Understanding Augmentations and Model Robustness

2025/05/30 by Yoo, Weebum, S.W. Yoon, Yoon, Sung Whan
Physics and Astronomy · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Model Reduction and Neural Networks

paper · pdf · doi:10.48550/arxiv.2505.24592

openalex publication_date 2025/05/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Model robustness indicates a model's capability to generalize well on unforeseen distributional shifts, including data corruptions and adversarial attacks. Data augmentation is one of the most prevalent and effective ways to enhance robustness. Despite the great success of the diverse augmentations in different fields, a unified theoretical understanding of their efficacy in improving model robustness is lacking. We theoretically reveal a general condition for label-preserving augmentations to bring robustness to diverse distribution shifts through the lens of flat minima and generalization bound, which de facto turns out to be strongly correlated with robustness against different distribution shifts in practice. Unlike most earlier works, our theoretical framework accommodates all the label-preserving augmentations and is not limited to particular distribution shifts. We substantiate our theories through different simulations on the existing common corruption and adversarial robustness benchmarks based on the CIFAR and ImageNet datasets.

Citations

• A Simple Background Augmentation Method for Object Detection with Diffusion Model
• DiffuseMix: Label-Preserving Data Augmentation with Diffusion Models
• Make Continual Learning Stronger via C-Flat
• On the Duality Between Sharpness-Aware Minimization and Adversarial Training
• A PAC-Bayesian Link Between Generalisation and Flat Minima
• Understanding Data Augmentation from a Robustness Perspective
• Chasing Fairness Under Distribution Shift: A Model Weight Perturbation Approach
• Data Augmentation Alone Can Improve Adversarial Training
• Rank Diminishing in Deep Neural Networks
• When Do Flat Minima Optimizers Work?
• Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscape
• PixMix: Dreamlike Pictures Comprehensively Improve Safety Measures
• Data Augmentation Can Improve Robustness
• Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minima
• AugMax: Adversarial Composition of Random Augmentations for Robust\n Training
• Fixing Data Augmentation to Improve Adversarial Robustness
• On Interaction Between Augmentations and Corruptions in Natural Corruption Robustness
• SWAD: Domain Generalization by Seeking Flat Minima
• Maximum-Entropy Adversarial Data Augmentation for Improved\n Generalization and Robustness
• Sharpness-Aware Minimization for Efficiently Improving Generalization
• In Search of Lost Domain Generalization
• The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
• Deep Domain-Adversarial Image Generation for Domain Generalisation
• Adversarial AutoAugment
• AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
• Fantastic Generalization Measures and Where to Find Them
• Faster AutoAugment: Learning Augmentation Strategies using\n Backpropagation
• RandAugment: Practical automated data augmentation with a reduced search space
• Towards Shape Biased Unsupervised Representation Learning for Domain Generalization
• A Fourier Perspective on Model Robustness in Computer Vision
• Are Labels Required for Improving Adversarial Robustness?
• Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks
• CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features
• Robustness to Adversarial Perturbations in Learning from Incomplete Data
• Fast AutoAugment
• Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
• Wide neural networks of any depth evolve as linear models under gradient descent *
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Style Augmentation: Data Augmentation via Style Randomization
• A Convergence Theory for Deep Learning via Over-Parameterization
• Learning Overparameterized Neural Networks via Stochastic Gradient\n Descent on Structured Data
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Generalizing to Unseen Domains via Adversarial Data Augmentation
• BAGAN: Data Augmentation with Balancing GAN
• Averaging Weights Leads to Wider Optima and Better Generalization
• mixup: Beyond Empirical Risk Minimization
• Improved Regularization of Convolutional Neural Networks with Cutout
• Towards Deep Learning Models Resistant to Adversarial Attacks
• Hide-and-Seek: Forcing a Network to be Meticulous for Weakly-supervised\n Object and Action Localization
• Aggregated Residual Transformations for Deep Neural Networks
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp\n Minima
• A Boundary Tilting Persepective on the Phenomenon of Adversarial Examples
• Densely Connected Convolutional Networks
• Striving for Simplicity: The All Convolutional Net
• Conditional Generative Adversarial Nets
• Effective data generation for imbalanced learning using conditional generative adversarial networks

Related

• Certifying Model Accuracy under Distribution Shifts
• Label Smoothing and Adversarial Robustness
• Label Augmentation for Neural Networks Robustness
• A Fourier Perspective on Model Robustness in Computer Vision
• The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization