Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization

2021/06/11 by Kartik Ahuja, Ahuja, Kartik, Ethan Caballero +11 · 31 citations
Computer Science · #Adversarial Robustness in Machine Learning #Bayesian Modeling and Causal Inference #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms

paper · pdf · doi:10.48550/arxiv.2106.06607

openalex publication_date 2021/06/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The invariance principle from causality is at the heart of notable approaches such as invariant risk minimization (IRM) that seek to address out-of-distribution (OOD) generalization failures. Despite the promising theory, invariance principle-based approaches fail in common classification tasks, where invariant (causal) features capture all the information about the label. Are these failures due to the methods failing to capture the invariance? Or is the invariance principle itself insufficient? To answer these questions, we revisit the fundamental assumptions in linear regression tasks, where invariance-based approaches were shown to provably generalize OOD. In contrast to the linear regression tasks, we show that for linear classification tasks we need much stronger restrictions on the distribution shifts, or otherwise OOD generalization is impossible. Furthermore, even with appropriate restrictions on distribution shifts in place, we show that the invariance principle alone is insufficient. We prove that a form of the information bottleneck constraint along with invariance helps address key failures when invariant features capture all the information about the label and also retains the existing success when they do not. We propose an approach that incorporates both of these principles and demonstrate its effectiveness in several experiments.

Cited by

• Contrast transfer functions help quantify neural network out-of-distribution generalization in HRTEM
• PISA: Prioritized Invariant Subgraph Aggregation
• Information-Theoretic Greedy Layer-wise Training for Traffic Sign Recognition
• Exploring Scale Shift in Crowd Localization under the Context of Domain Generalization
• Measure-Theoretic Anti-Causal Representation Learning
• Tackling Time-Series Forecasting Generalization via Mitigating Concept Drift
• Towards Generalizable PDE Dynamics Forecasting via Physics-Guided Invariant Learning
• Feed Two Birds with One Scone: Exploiting Function-Space Regularization for Both OOD Robustness and ID Fine-Tuning Performance
• Label Smoothing is a Pragmatic Information Bottleneck
• Class Unbiasing for Generalization in Medical Diagnosis
• Rethink Domain Generalization in Heterogeneous Sequence MRI Segmentation
• Should Bias Always be Eliminated? A Principled Framework to Use Data Bias for OOD Generation
• Group-wise Scaling and Orthogonal Decomposition for Domain-Invariant Feature Extraction in Face Anti-Spoofing
• Moment Alignment: Unifying Gradient and Hessian Matching for Domain Generalization
• Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains
• Pruning Spurious Subgraphs for Graph Out-of-Distribution Generalization
• Out-of-Distribution Graph Models Merging
• Towards Explicit Geometry-Reflectance Collaboration for Generalized LiDAR Segmentation in Adverse Weather
• Data Heterogeneity Modeling for Trustworthy Machine Learning
• Invariant Link Selector for Spatial-Temporal Out-of-Distribution Problem
• Bridging Distribution Shift and AI Safety: Conceptual and Methodological Synergies
• The Third Pillar of Causal Analysis? A Measurement Perspective on Causal Representations
• Robust Invariant Representation Learning by Distribution Extrapolation
• Mitigating Spurious Correlations with Causal Logit Perturbation
• Towards Comprehensive and Prerequisite-Free Explainer for Graph Neural Networks
• Unsupervised Invariant Risk Minimization
• Fine-Grained Class-Conditional Distribution Balancing for Debiased Learning
• BOOM: Benchmarking Out-Of-distribution Molecular Property Predictions of Machine Learning Models
• Conditionally Identifiable Latent-Environment Modeling for Out-of-Distribution Recommendation
• Class-Conditional Distribution Balancing for Group Robust Classification
• CANet: ChronoAdaptive Network for Enhanced Long-Term Time Series Forecasting under Non-Stationarity

Related

• Balancing Fairness and Robustness via Partial Invariance
• When is invariance useful in an Out-of-Distribution Generalization problem ?
• The Missing Invariance Principle Found -- the Reciprocal Twin of Invariant Risk Minimization
• The Risks of Invariant Risk Minimization
• Nonlinear Invariant Risk Minimization: A Causal Approach