Out of Distribution Generalization in Machine Learning

2021/03/03 by Martín Arjovsky, Arjovsky, Martin · 5 citations
Computer Science · #Bayesian Methods and Mixture Models #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms

paper · pdf · doi:10.48550/arxiv.2103.02667

openalex publication_date 2021/03/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Machine learning has achieved tremendous success in a variety of domains in recent years. However, a lot of these success stories have been in places where the training and the testing distributions are extremely similar to each other. In everyday situations when models are tested in slightly different data than they were trained on, ML algorithms can fail spectacularly. This research attempts to formally define this problem, what sets of assumptions are reasonable to make in our data and what kind of guarantees we hope to obtain from them. Then, we focus on a certain class of out of distribution problems, their assumptions, and introduce simple algorithms that follow from these assumptions that are able to provide more reliable generalization. A central topic in the thesis is the strong link between discovering the causal structure of the data, finding features that are reliable (when using them to predict) regardless of their context, and out of distribution generalization.

Citations

• Artificial intelligence: a modern approach
• Sequence to Sequence Learning with Neural Networks
• Estimating causal effects of treatments in randomized and nonrandomized studies.
• Equality of Opportunity in Supervised Learning
• Neural Architecture Search with Reinforcement Learning
• Certifying Some Distributional Robustness with Principled Adversarial\n Training
• Deep Learning: A Critical Appraisal
• Invariant Risk Minimization
• Learning dexterous in-hand manipulation

Cited by

• Invariant Feature Extraction Through Conditional Independence and the Optimal Transport Barycenter Problem: the Gaussian case
• Fairness Is Not Just Ethical: Performance Trade-Off via Data Correlation Tuning to Mitigate Bias in ML Software
• Distributionally Robust Optimization via Diffusion Ambiguity Modeling
• Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advances
• Optimization-based Causal Estimation from Heterogenous Environments

Related

• Towards Out-Of-Distribution Generalization: A Survey
• Towards a Theoretical Framework of Out-of-Distribution Generalization
• A Survey on Evaluation of Out-of-Distribution Generalization
• Separating Geometry from Probability in the Analysis of Generalization
• Distributional Generalization: A New Kind of Generalization