Mitigating Shortcut Learning with InterpoLated Learning

2025/07/07 by Michalis Korakakis, Andreas Vlachos, Korakakis, Michalis +3
Computer Science · #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2507.05527

openalex publication_date 2025/07/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Empirical risk minimization (ERM) incentivizes models to exploit shortcuts, i.e., spurious correlations between input attributes and labels that are prevalent in the majority of the training data but unrelated to the task at hand. This reliance hinders generalization on minority examples, where such correlations do not hold. Existing shortcut mitigation approaches are model-specific, difficult to tune, computationally expensive, and fail to improve learned representations. To address these issues, we propose InterpoLated Learning (InterpoLL) which interpolates the representations of majority examples to include features from intra-class minority examples with shortcut-mitigating patterns. This weakens shortcut influence, enabling models to acquire features predictive across both minority and majority examples. Experimental results on multiple natural language understanding tasks demonstrate that InterpoLL improves minority generalization over both ERM and state-of-the-art shortcut mitigation methods, without compromising accuracy on majority examples. Notably, these gains persist across encoder, encoder-decoder, and decoder-only architectures, demonstrating the method's broad applicability.

Citations

• Don't blame Dataset Shift! Shortcut Learning due to Gradients and Cross Entropy
• Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentation
• Think Twice: Measuring the Efficiency of Eliminating Prediction Shortcuts of Question Answering Models
• Change is Hard: A Closer Look at Subpopulation Shift
• Which Shortcut Solution Do Question Answering Models Prefer to Learn?
• GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspective
• Debiasing Masks: A New Framework for Shortcut Mitigation in NLU
• On Feature Learning in the Presence of Spurious Correlations
• Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations
• Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasets
• Augmenting Document Representations for Dense Retrieval with\n Interpolation and Perturbation
• On the Calibration of Pre-trained Language Models using Mixup Guided by\n Area Under the Margin and Saliency
• Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlations
• Improving Out-of-Distribution Robustness via Selective Augmentation
• Debiasing Methods in Natural Language Understanding Make Bias More\n Accessible
• Don't Discard All the Biased Instances: Investigating a Core Assumption in Dataset Bias Mitigation Techniques
• Just Train Twice: Improving Group Robustness without Training Group\n Information
• Fairness via Representation Neutralization
• SSMix: Saliency-Based Span Mixup for Text Classification
• Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU Models
• MixUp Training Leads to Reduced Overfitting and Improved Calibration for the Transformer Architecture
• Robustness to Spurious Correlations in Text Classification via Automatically Generated Counterfactuals
• WILDS: A Benchmark of in-the-Wild Distribution Shifts
• Learning from others' mistakes: Avoiding dataset biases without modeling them
• Towards Debiasing NLU Models from Unknown Biases
• Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics
• An Investigation of Why Overparameterization Exacerbates Spurious Correlations
• Mind the Trade-off: Debiasing NLU Models without Degrading the\n In-distribution Performance
• Syntactic Data Augmentation Increases Robustness to Inference Heuristics
• Shortcut learning in deep neural networks
• Information-Theoretic Probing with Minimum Description Length
• Adversarial Filters of Dataset Biases
• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• End-to-End Bias Mitigation by Modelling Biases in Corpora
• Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases
• Unlearn Dataset Bias in Natural Language Inference by Fitting the Residual
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• On Adversarial Removal of Hypothesis-only Bias in Natural Language Inference
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Augmenting Data with Mixup for Sentence Classification: An Empirical Study
• Nuanced Metrics for Measuring Unintended Bias with Real Data for Text\n Classification
• Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
• MixUp as Locally Linear Out-Of-Manifold Regularization
• Reducing Gender Bias in Abusive Language Detection
• Stress Test Evaluation for Natural Language Inference
• Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods
• FEVER: a large-scale dataset for Fact Extraction and VERification
• Annotation Artifacts in Natural Language Inference Data
• Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior
• mixup: Beyond Empirical Risk Minimization
• Adversarial Examples for Evaluating Reading Comprehension Systems
• A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• MiMu: Mitigating Multiple Shortcut Learning Behavior of Transformers
• A Typology for Exploring the Mitigation of Shortcut Behavior
• Don't blame Dataset Shift! Shortcut Learning due to Gradients and Cross Entropy
• Shortcut Learning in In-Context Learning: A Survey
• Mitigating Shortcuts in Language Models with Soft Label Encoding