RNNs as psycholinguistic subjects: Syntactic state and grammatical\n dependency

2018/09/05 by Richard Futrell, Futrell, Richard, Ethan Wilcox +5 · 4 citations
Computer Science · #Topic Modeling #Natural Language Processing Techniques #Text Readability and Simplification

paper · pdf · doi:10.48550/arxiv.1809.01329

Abstract

Recurrent neural networks (RNNs) are the state of the art in sequence\nmodeling for natural language. However, it remains poorly understood what\ngrammatical characteristics of natural language they implicitly learn and\nrepresent as a consequence of optimizing the language modeling objective. Here\nwe deploy the methods of controlled psycholinguistic experimentation to shed\nlight on to what extent RNN behavior reflects incremental syntactic state and\ngrammatical dependency representations known to characterize human linguistic\nbehavior. We broadly test two publicly available long short-term memory (LSTM)\nEnglish sequence models, and learn and test a new Japanese LSTM. We demonstrate\nthat these models represent and maintain incremental syntactic state, but that\nthey do not always generalize in the same way as humans. Furthermore, none of\nour models learn the appropriate grammatical dependency configurations\nlicensing reflexive pronouns or negative polarity items.\n

Citations

• Expectation-based syntactic comprehension

Cited by

• A Systematic Assessment of Syntactic Generalization in Neural Language Models
• Probing the phonetic and phonological knowledge of tones in Mandarin TTS models
• On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior
• Can RNNs learn Recursive Nested Subject-Verb Agreements?

Related

• How much complexity does an RNN architecture need to learn syntax-sensitive dependencies?
• Neural Language Models as Psycholinguistic Subjects: Representations of\n Syntactic State
• What Syntactic Structures block Dependencies in RNN Language Models?
• Can RNNs learn Recursive Nested Subject-Verb Agreements?
• On learning an interpreted language with recurrent models