Probing What Different NLP Tasks Teach Machines about Function Word Comprehension

2019/04/25 by Najoung Kim, Roma Patel, Kim, Najoung +21 · 2 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1904.11544

openalex publication_date 2019/04/25 · openalex created_date 2019/07/12 · openalex updated_date 2026/07/28

Abstract

We introduce a set of nine challenge tasks that test for the understanding of function words. These tasks are created by structurally mutating sentences from existing datasets to target the comprehension of specific types of function words (e.g., prepositions, wh-words). Using these probing tasks, we explore the effects of various pretraining objectives for sentence encoders (e.g., language modeling, CCG supertagging and natural language inference (NLI)) on the learned representations. Our results show that pretraining on language modeling performs the best on average across our probing tasks, supporting its widespread use for pretraining state-of-the-art NLP models, and CCG supertagging and NLI pretraining perform comparably. Overall, no pretraining objective dominates across the board, and our function word probing tasks highlight several intuitive differences between pretraining objectives, e.g., that NLI helps the comprehension of negation.

Citations

• Skip-Thought Vectors
• Long Short-Term Memory
• Bidirectional Attention Flow for Machine Comprehension
• The Natural Language Decathlon: Multitask Learning as Question Answering

Cited by

• Behavior and Representation in Large Language Models for Combinatorial Optimization: From Feature Extraction to Algorithm Selection
• BeDiscovER: The Benchmark of Discourse Understanding in the Era of Reasoning Language Models
• TRACEALIGN -- Tracing the Drift: Attributing Alignment Failures to Training-Time Belief Sources in LLMs

Related

• Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?
• Bag of Tricks for Effective Language Model Pretraining and Downstream Adaptation: A Case Study on GLUE
• Negation, Coordination, and Quantifiers in Contextualized Language Models
• Does He Wink or Does He Nod? A Challenging Benchmark for Evaluating Word Understanding of Language Models
• Subspace Chronicles: How Linguistic Information Emerges, Shifts and Interacts during Language Model Training