A Group-Theoretic Framework for Data Augmentation

2019/07/25 by Shuxiao Chen, Chen, Shuxiao, Edgar Dobriban +4
Chemistry · Computer Science · Mathematics · #Chemistry #Computer science #FOS: Computer and information sciences #FOS: Mathematics #Group (periodic table) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Privacy-Preserving Technologies in Data #Statistics Theory (math.ST) #Stochastic Gradient Optimization Techniques #cs.LG #math.ST #stat.ML #stat.TH

paper · pdf · doi:10.48550/arxiv.1907.10905

To appear in Journal of Machine Learning Research

openalex publication_date 2019/07/25 · arxiv created 2020/11/06 · arxiv updated 2020/11/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Data augmentation is a widely used trick when training deep neural networks: in addition to the original data, properly transformed data are also added to the training set. However, to the best of our knowledge, a clear mathematical framework to explain the performance benefits of data augmentation is not available. In this paper, we develop such a theoretical framework. We show data augmentation is equivalent to an averaging operation over the orbits of a certain group that keeps the data distribution approximately invariant. We prove that it leads to variance reduction. We study empirical risk minimization, and the examples of exponential families, linear regression, and certain two-layer neural networks. We also discuss how data augmentation could be used in problems with symmetry where other approaches are prevalent, such as in cryo-electron microscopy (cryo-EM).

Citations

• Deep Learning in Neural Networks: An Overview
• Conditional Generative Adversarial Nets
• Deep, Big, Simple Neural Nets for Handwritten Digit Recognition
• Backpropagation Applied to Handwritten Zip Code Recognition
• Improved Regularization of Convolutional Neural Networks with Cutout
• Random Erasing Data Augmentation
• mixup: Beyond Empirical Risk Minimization
• Data Augmentation Generative Adversarial Networks
• Estimation under group actions: Recovering orbits from invariants
• On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups
• Data augmentation instead of explicit regularization
• AutoAugment: Learning Augmentation Policies from Data
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• The Complexity of Making the Gradient Small in Stochastic Convex Optimization
• Augment your batch: better training with larger batches
• Bad Global Minima Exist and SGD Can Reach Them
• Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation
• Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules
• Unsupervised Data Augmentation for Consistency Training
• How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?
• Steerable ePCA: Rotationally Invariant Exponential Family PCA
• Learning Multiple Layers of Features from Tiny Images

Related

• Affinity and Diversity: Quantifying Mechanisms of Data Augmentation
• Data Augmentation Revisited: Rethinking the Distribution Gap between Clean and Augmented Data
• Data Augmentation and Regularization for Learning Group Equivariance
• On the Benefits of Invariance in Neural Networks
• Ensembles provably learn equivariance through data augmentation