Diverging Transformer Predictions for Human Sentence Processing: A Comprehensive Analysis of Agreement Attraction Effects

2026/03/17 by Titus von der Malsburg, Sebastian Padó · 1 voice
Computer Science · Neuroscience · #Autoregressive model #Cognition #Natural Language Processing Techniques #Neurobiology of Language and Bilingualism #Phrase #Sentence #Sentence processing #Spurious relationship #Topic Modeling #Transformer #cs.CL

paper · pdf · open access · doi:10.48550/arxiv.2603.16574

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2026/03/17 · arxiv published 2026/03/17 · arxiv updated 2026/03/17 · openalex created_date 2026/03/20 · openalex updated_date 2026/07/28

Abstract

Transformers underlie almost all state-of-the-art language models in computational linguistics, yet their cognitive adequacy as models of human sentence processing remains disputed. In this work, we use a surprisal-based linking mechanism to systematically evaluate eleven autoregressive transformers of varying sizes and architectures on a more comprehensive set of English agreement attraction configurations than prior work. Our experiments yield mixed results: While transformer predictions generally align with human reading time data for prepositional phrase configurations, performance degrades significantly on object-extracted relative clause configurations. In the latter case, predictions also diverge markedly across models, and no model successfully replicates the asymmetric interference patterns observed in humans. We conclude that current transformer models do not explain human morphosyntactic processing, and that evaluations of transformers as cognitive models must adopt rigorous, comprehensive experimental designs to avoid spurious generalizations from isolated syntactic configurations or individual models.

Citations

• Causal Estimation of Tokenisation Bias
• A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processing
• Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?
• Accounting for Agreement Phenomena in Sentence Comprehension with Transformer Language Models: Effects of Similarity-based Interference on Surprisal and Attention
• On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior
• Language Models are Few-Shot Learners
• Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)
• Distinct patterns of syntactic agreement errors in recurrent networks and humans
• Colorless green recurrent networks dream hierarchically
• Attention Is All You Need
• Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
• Expectation-based syntactic comprehension
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Discussions

• New preprint with Sebastian Padó: Transformers do not reliably explain human morphosyntactic processing. Testing 11 models on agreement attraction shows they match human data for prepositional modifie [bsky, 1 points, 0 comments]

Related

• Accounting for Agreement Phenomena in Sentence Comprehension with Transformer Language Models: Effects of Similarity-based Interference on Surprisal and Attention
• Assessing the Capacity of Transformer to Abstract Syntactic Representations: A Contrastive Analysis Based on Long-distance Agreement
• Are Transformers a Modern Version of ELIZA? Observations on French Object Verb Agreement
• Explaining How Transformers Use Context to Build Predictions
• Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metrics