From Invariant Representations to Invariant Data: Provable Robustness to Spurious Correlations via Noisy Counterfactual Matching

2025/05/30 by Ruqi Bai, Ji Yao, Bai, Ruqi +5 · 1 citation
Computer Science · #Bayesian Modeling and Causal Inference #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2505.24843

openalex publication_date 2025/05/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Models that learn spurious correlations from training data often fail when deployed in new environments. While many methods aim to learn invariant representations to address this, they often underperform standard empirical risk minimization (ERM). We propose a data-centric alternative that shifts the focus from learning invariant representations to leveraging invariant data pairs -- pairs of samples that should have the same prediction. We prove that certain counterfactuals naturally satisfy this invariance property. Based on this, we introduce Noisy Counterfactual Matching (NCM), a simple constraint-based method that improves robustness by leveraging even a small number of noisy counterfactual pairs -- improving upon prior works that do not explicitly consider noise. For linear causal models, we prove that NCM's test-domain error is bounded by its in-domain error plus a term dependent on the counterfactuals' quality and diversity. Experiments on synthetic data validate our theory, and we demonstrate NCM's effectiveness on real-world datasets.

Citations

• On the Foundations of Shortcut Learning
• Identifiability Guarantees for Causal Disentanglement from Soft Interventions
• Test-Time Training on Video Streams
• Benchmarking Algorithms for Federated Domain Generalization
• Towards Characterizing Domain Counterfactuals For Invertible Latent Causal Models
• Nonparametric Identifiability of Causal Representations from Unknown Interventions
• Improving Domain Generalization with Domain Relations
• Aggregation of Disentanglement: Reconsidering Domain Variations in Domain Generalization
• Identifying Weight-Variant Latent Causal Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• Toward Causal Representation Learning
• Gradient Matching for Domain Generalization
• Learning Transferable Visual Models From Natural Language Supervision
• Domain Invariant Representation Learning with Domain Density Transformations
• WILDS: A Benchmark of in-the-Wild Distribution Shifts
• The Risks of Invariant Risk Minimization
• In Search of Lost Domain Generalization
• Domain Generalization using Causal Matching
• Out-of-Distribution Generalization via Risk Extrapolation (REx)
• Calibrating Deep Neural Networks using Focal Loss
• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
• Generalizing to unseen domains via distribution matching
• Hidden Stratification Causes Clinically Meaningful Failures in Machine\n Learning for Medical Imaging
• Invariant Risk Minimization
• Multi-component Image Translation for Deep Domain Generalization
• ComboGAN: Unrestrained Scalability for Image Domain Translation
• SchNet – A deep learning architecture for molecules and materials
• Domain Generalization by Marginal Transfer Learning
• Learning to Generalize: Meta-Learning for Domain Generalization
• Deeper, Broader and Artier Domain Generalization
• Deep CORAL: Correlation Alignment for Deep Domain Adaptation
• Deep Residual Learning for Image Recognition
• Causal inference using invariant prediction: identification and confidence intervals
• Adam: A Method for Stochastic Optimization
• Domain Adaptation: Learning Bounds and Algorithms
• Perturbation bounds in connection with singular value decomposition

Cited by

• Domain Generalization Under Posterior Drift

Related

• Counterfactual Invariance to Spurious Correlations: Why and How to Pass Stress Tests
• Invariant Risk Minimization
• Exogenous Matching: Learning Good Proposals for Tractable Counterfactual Estimation
• Nonlinear Invariant Risk Minimization: A Causal Approach
• Don't Explain Noise: Robust Counterfactuals for Randomized Ensembles