2021/09/21 by Bingzhi Li, Li, Bingzhi, Guillaume Wisniewski +4
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling #cs.CL
paper · pdf · doi:10.48550/arxiv.2109.10133
Camera-ready for EMNLP'21
arxiv created 2021/09/21 · openalex publication_date 2021/09/21 · arxiv updated 2021/09/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Many recent works have demonstrated that unsupervised sentence representations of neural networks encode syntactic information by observing that neural language models are able to predict the agreement between a verb and its subject. We take a critical look at this line of research by showing that it is possible to achieve high accuracy on this agreement task with simple surface heuristics, indicating a possible flaw in our assessment of neural networks' syntactic ability. Our fine-grained analyses of results on the long-range French object-verb agreement show that contrary to LSTMs, Transformers are able to capture a non-trivial amount of grammatical structure.