Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics

2021/10/04 by Bhargava, Prajjwal, Drozd, Aleksandr, Rogers, Anna · 4 citations
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2110.01518

Abstract

Much of recent progress in NLU was shown to be due to models' learning dataset-specific heuristics. We conduct a case study of generalization in NLI (from MNLI to the adversarially constructed HANS dataset) in a range of BERT-based architectures (adapters, Siamese Transformers, HEX debiasing), as well as with subsampling the data and increasing the model size. We report 2 successful and 3 unsuccessful strategies, all providing insights into how Transformer-based models learn to generalize.

Cited by

• Do Generalisation Results Generalise?
• WARBERT: A Hierarchical BERT-based Model for Web API Recommendation
• DACTYL: Diverse Adversarial Corpus of Texts Yielded from Large Language Models
• Understanding the Evolution of the Neural Tangent Kernel at the Edge of Stability

Related

• BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance
• Out-of-distribution generalization via composition: a lens through induction heads in Transformers
• Out-of-distribution generalization via composition: A lens through induction heads in Transformers
• Diversify Your Datasets: Analyzing Generalization via Controlled Variance in Adversarial Datasets
• Uncovering More Shallow Heuristics: Probing the Natural Language Inference Capacities of Transformer-Based Pre-Trained Language Models Using Syllogistic Patterns