Invariant Risk Minimization

2019/07/05 by Martín Arjovsky, Arjovsky, Martin, Léon Bottou +5 · 181 citations
Computer Science · #Domain Adaptation and Few-Shot Learning #Machine Learning and Data Classification #Machine Learning and Algorithms

paper · pdf · doi:10.48550/arxiv.1907.02893

Abstract

We introduce Invariant Risk Minimization (IRM), a learning paradigm to estimate invariant correlations across multiple training distributions. To achieve this goal, IRM learns a data representation such that the optimal classifier, on top of that data representation, matches for all training distributions. Through theory and experiments, we show how the invariances learned by IRM relate to the causal structures governing the data and enable out-of-distribution generalization.

Citations

• Estimating causal effects of treatments in randomized and nonrandomized studies.

Cited by

• Let Samples Speak: Mitigating Spurious Correlation by Exploiting the Clusterness of Samples
• Every Client Is an Environment: Federated De-confounding for Spatio-Temporal Forecasting
• Entangled by Design: Spurious Intra-Variable Signal Routing in Tabular In-Context Learners
• From Training to Deployment: Post-Hoc Causal Feature Identification via Sensitivity Ratios
• On the Opportunities and Risks of Foundation Models
• Invariant Feature Extraction Through Conditional Independence and the Optimal Transport Barycenter Problem: the Gaussian case
• Clever Hans in Chemistry: Chemist Style Signals Confound Activity Prediction on Public Benchmarks
• Localising Shortcut Learning in Pixel Space via Ordinal Scoring Correlations for Attribution Representations (OSCAR)
• Domain Adaptation in Structural Health Monitoring of Civil Infrastructure: A Systematic Review
• CIRR: Causal-Invariant Retrieval-Augmented Recommendation with Faithful Explanations under Distribution Shift
• Unifying Causal Reinforcement Learning: Survey, Taxonomy, Algorithms and Applications
• Adversarially Robust Detection of Harmful Online Content: A Computational Design Science Approach
• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
• Gradient Starvation: A Learning Proclivity in Neural Networks
• Maximum Risk Minimization with Random Forests
• Federated Domain Generalization with Latent Space Inversion
• Self-Ensemble Post Learning for Noisy Domain Generalization
• Representation Invariance and Allocation: When Subgroup Balance Matters
• Causality-inspired Single-source Domain Generalization for Medical Image Segmentation
• Contrast transfer functions help quantify neural network out-of-distribution generalization in HRTEM
• Modality-Aware Bias Mitigation and Invariance Learning for Unsupervised Visible-Infrared Person Re-Identification
• Underspecification Presents Challenges for Credibility in Modern Machine Learning
• MIDG: Mixture of Invariant Experts with knowledge injection for Domain Generalization in Multimodal Sentiment Analysis
• Learning Invariant Graph Representations Through Redundant Information
• Stable Single-Pixel Contrastive Learning for Semantic and Geometric Tasks
• Learning Causal Semantic Representation for Out-of-Distribution Prediction
• HydroDCM: Hydrological Domain-Conditioned Modulation for Cross-Reservoir Inflow Prediction
• Graph Data Augmentation with Contrastive Learning on Covariate Distribution Shift
• A Causal View on Robustness of Neural Networks
• Stable predictions for health related anticausal prediction tasks affected by selection biases: the need to deconfound the test set features
• On Calibration and Out-of-domain Generalization
• PISA: Prioritized Invariant Subgraph Aggregation
• Causal Inference: A Tale of Three Frameworks
• A Probabilistic Framework for Temporal Distribution Generalization in Industry-Scale Recommender Systems
• Counterfactual Generative Networks
• The Impact of Off-Policy Training Data on Probe Generalisation
• On the Transfer of Disentangled Representations in Realistic Settings
• Proactive Pseudo-Intervention: Causally Informed Contrastive Learning For Interpretable Vision Models
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Selective Mixup for Debiasing Question Selection in Computerized Adaptive Testing
• Learning to See Through a Baby's Eyes: Early Visual Diets Enable Robust Visual Intelligence in Humans and Machines
• Predicting with High Correlation Features
• BREEDS: Benchmarks for Subpopulation Shift
• Gradient Matching for Domain Generalization
• Model Patching: Closing the Subgroup Performance Gap with Data Augmentation
• When is invariance useful in an Out-of-Distribution Generalization problem ?
• Causality-aware counterfactual confounding adjustment as an alternative to linear residualization in anticausal prediction tasks based on linear learners
• Did Models Sufficient Learn? Attribution-Guided Training via Subset-Selected Counterfactual Augmentation
• Generalizing to unseen domains via distribution matching
• Multi-Objective Bilevel Learning
• Improving Deepfake Detection with Reinforcement Learning-Based Adaptive Data Augmentation
• Peeling Context from Cause for Molecular Property Prediction
• CaberNet: Causal Representation Learning for Cross-Domain HVAC Energy Prediction
• MoRA: Missing Modality Low-Rank Adaptation for Visual Recognition
• Spurious Correlation-Aware Embedding Regularization for Worst-Group Robustness
• Causal Graph Neural Networks for Healthcare
• Evolving Graph Learning for Out-of-Distribution Generalization in Non-stationary Environments
• Self-Harmony: Learning to Harmonize Self-Supervision and Self-Play in Test-Time Reinforcement Learning
• Learning with Category-Equivariant Representations for Human Activity Recognition
• Community Detection on Model Explanation Graphs for Explainable AI
• ODP-Bench: Benchmarking Out-of-Distribution Performance Prediction
• On Measuring Localization of Shortcuts in Deep Networks
• Transferring Causal Effects using Proxies
• An Online Learning Approach to Interpolation and Extrapolation in Domain\n Generalization
• A Compression Perspective on Simplicity Bias
• Quantifying and Improving Transferability in Domain Generalization
• Algorithmic Bias and Data Bias: Understanding the Relation between Distributionally Robust Optimization and Data Curation
• Generalizing Graph Neural Networks on Out-Of-Distribution Graphs
• For Manifold Learning, Deep Neural Networks can be Locality Sensitive Hash Functions
• An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models
• G2VD: Generalizable AI-Generated Video Detection via Counterfactual Intervention and Causal Disentanglement
• Causal Attention for Unbiased Visual Recognition
• Causal Learning for Socially Responsible AI
• Mitigating Spurious Correlation via Distributionally Robust Learning with Hierarchical Ambiguity Sets
• Nonlinear Invariant Risk Minimization: A Causal Approach
• Focus on the Common Good: Group Distributional Robustness Follows
• Robust Classification under Class-Dependent Domain Shift
• Embracing the Dark Knowledge: Domain Generalization Using Regularized Knowledge Distillation
• The Connection between Out-of-Distribution Generalization and Privacy of ML Models
• Multi-label Iterated Learning for Image Classification with Label Ambiguity
• An Analysis of Causal Effect Estimation using Outcome Invariant Data Augmentation
• Towards Robust and Adaptive Motion Forecasting: A Causal Representation Perspective
• Aggregation Hides Out-of-Distribution Generalization Failures from Spurious Correlations
• SAND-mask: An Enhanced Gradient Masking Strategy for the Discovery of Invariances in Domain Generalization
• How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Data
• Automatic Assessment of Students' Classroom Engagement with Bias Mitigated Multi-task Model
• Preventing Shortcuts in Adapter Training via Providing the Shortcuts
• Environment Inference for Learning Generalizable Dynamical System
• Towards Single-Source Domain Generalized Object Detection via Causal Visual Prompts
• Exploring Scale Shift in Crowd Localization under the Context of Domain Generalization
• Rectifying Shortcut Behaviors in Preference-based Reward Learning
• Overinterpretation reveals image classification model pathologies
• Causal Analysis of Agent Behavior for AI Safety
• Unbiased Gradient Low-Rank Projection
• Connecting Domains and Contrasting Samples: A Ladder for Domain Generalization
• Causal Time Series Modeling of Supraglacial Lake Evolution in Greenland under Distribution Shift
• Measure-Theoretic Anti-Causal Representation Learning
• Tackling Time-Series Forecasting Generalization via Mitigating Concept Drift
• Linear Regression Games: Convergence Guarantees to Approximate Out-of-Distribution Solutions
• ESI: Epistemic Uncertainty Quantification via Semantic-preserving Intervention for Large Language Models
• A tutorial on discovering and quantifying the effect of latent causal sources of multimodal EHR data
• Universal Adaptive Environment Discovery
• MEASURE: Multi-scale Minimal Sufficient Representation Learning for Domain Generalization in Sleep Staging
• Variational Disentanglement for Domain Generalization
• FML-bench: A Benchmark for Automatic ML Research Agents Highlighting the Importance of Exploration Breadth
• Subspace Regularizers for Few-Shot Class Incremental Learning
• Training Feature Attribution for Vision Models
• Evaluating Model Robustness and Stability to Dataset Shift
• Revisiting Mixout: An Overlooked Path to Robust Finetuning
• High-Rate Mixout: Revisiting Mixout for Robust Domain Generalization
• LLM Unlearning Under the Microscope: A Full-Stack View on Methods and Metrics
• Bridging Reasoning to Learning: Unmasking Illusions using Complexity Out of Distribution Generalization
• Domain Generalization Under Posterior Drift
• Learning Invariant Representations and Risks for Semi-supervised Domain Adaptation
• In Search of Robust Measures of Generalization
• Learning explanations that are hard to vary
• Downgrade to Upgrade: Optimizer Simplification Enhances Robustness in LLM Unlearning
• Robust Context-Aware Object Recognition
• Few-shot Domain Adaptation by Causal Mechanism Transfer
• Causally Guided Gaussian Perturbations for Out-Of-Distribution Generalization in Medical Imaging
• How Does Preconditioning Guide Feature Learning in Deep Neural Networks?
• Towards Generalizable PDE Dynamics Forecasting via Physics-Guided Invariant Learning
• Fidelity-Aware Data Composition for Robust Robot Generalization
• Does Weak-to-strong Generalization Happen under Spurious Correlations?
• GroupCoOp: Group-robust Fine-tuning via Group Prompt Learning
• FedDAPL: Toward Client-Private Generalization in Federated Learning
• Temporal Generalization: A Reality Check
• Causally-Enhanced Reinforcement Policy Optimization
• Adaptive Invariance for Molecule Property Prediction
• Learning the Wrong Lessons: Syntactic-Domain Spurious Correlations in Language Models
• Efficient nonparametric statistical inference on population feature importance using Shapley values
• Continuous Coordination As a Realistic Scenario for Lifelong Learning
• MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answering
• Unsupervised Domain Adaptation for Binary Classification with an Unobservable Source Subpopulation
• Shortcut to Nowhere: Demystifying Deep Spurious Regression
• Causal Transfer Random Forest: Combining Logged Data and Randomized Experiments for Robust Prediction
• Selecting Data Augmentation for Simulating Interventions
• Saliency is a Possible Red Herring When Diagnosing Poor Generalization
• Learning Fragmentation Physics or Exploiting Sequence Priors? Benchmarking Bias in Deep Learning Models for De Novo Peptide Sequencing
• Uniform Priors for Data-Efficient Transfer
• An Information-theoretic Approach to Distribution Shifts
• Robustness through Data Augmentation Loss Consistency
• CoDoL: Conditional Domain Prompt Learning for Out-of-Distribution Generalization
• Self Identity Mapping
• Consistent View Alignment Improves Foundation Models for 3D Medical Image Segmentation
• Deceptive Risk Minimization: Out-of-Distribution Generalization by Deceiving Distribution Shift Detectors
• SWAD: Domain Generalization by Seeking Flat Minima
• PersonaX: Multimodal Datasets with LLM-Inferred Behavior Traits
• Retiring Adult: New Datasets for Fair Machine Learning
• Preserving Domain Generalization in Fine-Tuning via Joint Parameter Selection
• Shortcut learning in deep neural networks
• Does Invariant Risk Minimization Capture Invariance?
• Evaluation of Causal Structure Learning Algorithms via Risk Estimation
• ACE and Diverse Generalization via Selective Disagreement
• Out of Distribution Generalization in Machine Learning
• Invariant Risk Minimization Games
• Detecting Regional Spurious Correlations in Vision Transformers via Token Discarding
• Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization
• Single Domain Generalization in Diabetic Retinopathy: A Neuro-Symbolic Learning Approach
• WILDS: A Benchmark of in-the-Wild Distribution Shifts
• BM-CL: Bias Mitigation through the lens of Continual Learning
• Generalizing Nucleus Recognition Model in Multi-source Images via Pruning
• Towards Out-Of-Distribution Generalization: A Survey
• Face4FairShifts: A Large Image Benchmark for Fairness and Robust Learning across Visual Domains
• Quantum Entanglement as Super-Confounding: From Bell's Theorem to Robust Machine Learning
• ERIS: An Energy-Guided Feature Disentanglement Framework for Out-of-Distribution Time Series Classification
• In Search of Lost Domain Generalization
• Superclass-Guided Representation Disentanglement for Spurious Correlation Mitigation
• Partial Optimal Transport with Applications on Positive-Unlabeled\n Learning
• Defending Against Image Corruptions Through Adversarial Augmentations
• Class Unbiasing for Generalization in Medical Diagnosis
• From Explainable to Explained AI: Ideas for Falsifying and Quantifying Explanations
• Multimodal Causal-Driven Representation Learning for Generalizable Medical Image Segmentation
• Unified modality separation: A vision-language framework for unsupervised domain adaptation
• SPA++: Generalized Graph Spectral Alignment for Versatile Domain Adaptation
• CaPulse: Detecting Anomalies by Tuning in to the Causal Rhythms of Time Series
• I³-MRec: Invariant Learning with Information Bottleneck for Incomplete Modality Recommendation
• Uncertainty Sets for Distributionally Robust Bandits Using Structural Equation Models
• Invariant Graph Transformer for Out-of-Distribution Generalization
• Consensus-Driven Active Model Selection
• From LLMs to Edge: Parameter-Efficient Fine-Tuning on Edge Devices

Related

• The Risks of Invariant Risk Minimization
• An Empirical Study of Invariant Risk Minimization
• Heterogeneous Risk Minimization
• Learning Optimal Features via Partial Invariance
• Nonlinear Invariant Risk Minimization: A Causal Approach