Assessing BERT's Syntactic Abilities

2019/01/16 by Goldberg, Yoav · 5 citations
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.1901.05287

Abstract

I assess the extent to which the recently introduced BERT model captures English syntactic phenomena, using (1) naturally-occurring subject-verb agreement stimuli; (2) "coloreless green ideas" subject-verb agreement stimuli, in which content words in natural sentences are randomly replaced with words sharing the same part-of-speech and inflection; and (3) manually crafted stimuli for subject-verb agreement and reflexive anaphora phenomena. The BERT model performs remarkably well on all cases.

Cited by

• LingGym: How Far Are LLMs from Thinking Like Field Linguists?
• Disaggregation Reveals Hidden Training Dynamics: The Case of Agreement Attraction
• Training-Free Spectral Fingerprints of Voice Processing in Transformers
• Steering Prepositional Phrases in Language Models: A Case of with-headed Adjectival and Adverbial Complements in Gemma-2
• Identifying a Circuit for Verb Conjugation in GPT-2

Related

• Inducing Syntactic Trees from BERT Representations
• Investigating BERT's Knowledge of Language: Five Analysis Methods with\n NPIs
• Subject Verb Agreement Error Patterns in Meaningless Sentences: Humans vs. BERT
• Open Sesame: Getting Inside BERT's Linguistic Knowledge
• Does BERT agree? Evaluating knowledge of structure dependence through agreement relations