Filtered Corpus Training (FiCT) Shows that Language Models Can Generalize from Indirect Evidence

2024/01/01 by Abhinav Patil, Jaap Jumelet, Yu Ying Chiu +5 · 1 voice · 13 citations
Computer Science · #Topic Modeling #Natural Language Processing Techniques #Speech Recognition and Synthesis

paper · doi:10.1162/tacl_a_00720

Abstract

Abstract This paper introduces Filtered Corpus Training, a method that trains language models (LMs) on corpora with certain linguistic constructions filtered out from the training data, and uses it to measure the ability of LMs to perform linguistic generalization on the basis of indirect evidence. We apply the method to both LSTM and Transformer LMs (of roughly comparable size), developing filtered corpora that target a wide range of linguistic phenomena. Our results show that while transformers are better qua LMs (as measured by perplexity), both models perform equally and surprisingly well on linguistic generalization measures, suggesting that they are capable of generalizing from indirect evidence.

Citations

• Expectation-based syntactic comprehension

Cited by

• Exposure is Optional: Learning Unlike Coordination in Language Models
• Collocational bootstrapping: A hypothesis about the learning of subject-verb agreement in humans and neural networks
• Not-So-Strange Love: Language Models and Generative Linguistic Theories are More Compatible than They Appear
• Across the Levels of Analysis: Explaining Predictive Processing in Humans Requires More Than Machine-Estimated Probabilities
• Cross-Modal Taxonomic Generalization in (Vision-) Language Models
• Linguists should learn to love speech-based deep learning models
• You can’t fight in here! This is BBS!
• Both Direct and Indirect Evidence Contribute to Dative Alternation Preferences in Language Models
• How linguistics learned to stop worrying and love the language models
• Targeted Linguistic Analysis of Sign Language Models with Minimal Translation Pairs
• Large language models are not about natural language
• Are Language Models Models?
• The Structural Sources of Verb Meaning Revisited: Large Language Models Display Syntactic Bootstrapping

Discussions

• Excited that the final version of our paper on filtered corpus training is out now in TACL! (Original thread is on the bad site, but the paper's title is a succinct summary.) doi.org/10.1162/tacl... [bsky, 17 points, 1 comments]

Related

• Filtered Corpus Training (FiCT) Shows that Language Models can Generalize from Indirect Evidence
• How Abstract Is Linguistic Generalization in Large Language Models? Experiments with Argument Structure
• Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMs
• How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biases
• What do Large Language Models Learn beyond Language?