Assessing BERT's Syntactic Abilities

2019/01/16 by Yoav Goldberg, Goldberg, Yoav · 17 citations
Computer Science · Neuroscience · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Neurobiology of Language and Bilingualism #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.1901.05287

arxiv created 2019/01/16 · openalex publication_date 2019/01/16 · arxiv updated 2019/01/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

I assess the extent to which the recently introduced BERT model captures English syntactic phenomena, using (1) naturally-occurring subject-verb agreement stimuli; (2) "coloreless green ideas" subject-verb agreement stimuli, in which content words in natural sentences are randomly replaced with words sharing the same part-of-speech and inflection; and (3) manually crafted stimuli for subject-verb agreement and reflexive anaphora phenomena. The BERT model performs remarkably well on all cases.

Cited by

• LingGym: How Far Are LLMs from Thinking Like Field Linguists?
• Disaggregation Reveals Hidden Training Dynamics: The Case of Agreement Attraction
• Training-Free Spectral Fingerprints of Voice Processing in Transformers
• Steering Prepositional Phrases in Language Models: A Case of with-headed Adjectival and Adverbial Complements in Gemma-2
• Identifying a Circuit for Verb Conjugation in GPT-2
• 75 Languages, 1 Model: Parsing Universal Dependencies Universally
• Have Attention Heads in BERT Learned Constituency Grammar?
• What Does BERT Look at? An Analysis of BERT’s Attention
• Trait-space Monitoring for Emergent Misalignment During Supervised Finetuning
• Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
• Heads-up! Unsupervised Constituency Parsing via Self-Attention Heads
• A Relation-Oriented Clustering Method for Open Relation Extraction
• Language Models Generalize to Human-like Word Order Preferences
• Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals
• TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
• Exploring Compositional Generalization (in COGS/ReCOGS_pos) by Transformers using Restricted Access Sequence Processing (RASP)
• Improving BERT Pretraining with Syntactic Supervision
• Investigating Syntactic Biases in Multilingual Transformers with RC Attachment Ambiguities in Italian and English

Related

• Inducing Syntactic Trees from BERT Representations
• Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIs
• Subject Verb Agreement Error Patterns in Meaningless Sentences: Humans vs. BERT
• Open Sesame: Getting Inside BERT's Linguistic Knowledge
• Does BERT agree? Evaluating knowledge of structure dependence through\n agreement relations