The neural architecture of language: Integrative modeling converges on predictive processing

2021/11/04 by Martin Schrimpf, Idan Blank, Greta Tuckute +5 · 1 voice · 21 citations
Computer Science · Neuroscience · Psychology · #Action Observation and Synchronization #Neurobiology of Language and Bilingualism #Topic Modeling

paper · doi:10.1073/pnas.2105646118

openalex publication_date 2021/11/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

The neuroscience of perception has recently been revolutionized with an integrative modeling approach in which computation, brain function, and behavior are linked across many datasets and many computational models. By revealing trends across models, this approach yields novel insights into cognitive and neural mechanisms in the target domain. We here present a systematic study taking this approach to higher-level cognition: human language processing, our species' signature cognitive skill. We find that the most powerful "transformer" models predict nearly 100% of explainable variance in neural responses to sentences and generalize across different datasets and imaging modalities (functional MRI and electrocorticography). Models' neural fits ("brain score") and fits to behavioral responses are both strongly correlated with model accuracy on the next-word prediction task (but not other language tasks). Model architecture appears to substantially contribute to neural fit. These results provide computationally explicit evidence that predictive processing fundamentally shapes the language comprehension mechanisms in the human brain.

Citations

• Incremental interpretation at verbs: restricting the domain of subsequent reference
• Predictive coding: a fresh view of inhibition in the retina
• Linguistic complexity: locality of syntactic dependencies
• Learning and development in neural networks: the importance of starting small
• ImageNet classification with deep convolutional neural networks
• Expectation-based syntactic comprehension
• Skip-Thought Vectors
• On the computational architecture of the neocortex
• The anatomy of language: a review of 100 fMRI studies published in 2009
• Finding Structure in Time
• Whatever next? Predictive brains, situated agents, and the future of cognitive science
• Exploring the Limits of Language Modeling
• Pointer Sentinel Mixture Models
• Deep Learning: A Critical Appraisal
• Neural population control via deep image synthesis
• Cross-lingual Language Model Pretraining
• Quasi-Recurrent Neural Networks
• Brain-Like Object Recognition with High-Performing Shallow Recurrent\n ANNs
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior

Cited by

• The language network responds robustly to sentences across tasks
• Characterizing semantic compositions in the brain: A model-driven fMRI re-analysis
• Deep neural networks and humans both benefit from compositional language structure
• Aligning statistical models with inference goals in the neuroscience of language: A dual-dependency taxonomy
• Neurobiological Causal Models of Language Processing
• Specifying precision in visual-orthographic prediction error representations for a better understanding of efficient reading
• Deep problems with neural network models of human vision
• On the importance of severely testing deep learning models of cognition
• Revealing emergent human-like conceptual representations from language prediction
• A systematic evaluation of Dutch large language models’ surprisal estimates in sentence, paragraph and book reading
• A shared model-based linguistic space for transmitting our thoughts from brain to brain in natural conversations
• Beyond computational equivalence: the behavioral inference principle for machine consciousness
• Language is primarily a tool for communication rather than thought
• Toward Complementary Intelligence: Integrating Cognitive and Machine AI
• ROSE: A Universal Neural Grammar
• Large language models can segment narrative events similarly to humans
• A gradient of complementary learning systems emerges through meta-learning
• Divergences between Language Models and Human Brains
• Bayesian Surprise Predicts Human Event Segmentation in Story Listening
• Linguistic Structure from a Bottleneck on Sequential Information Processing
• Contextual feature extraction hierarchies converge in large language models and the brain

Discussions

• In 2021 we were surprised to find that untrained language models are already decent predictors of activity in the human language system (doi.org/10.1073/pnas...). In arxiv.org/abs/2406.15109, we iden [bsky, 9 points, 1 comments]

Related

• Do Large Language Models Think Like the Brain? Sentence-Level Evidences from Layer-Wise Embeddings and fMRI
• Predictive Coding or Just Feature Discovery? An Alternative Account of Why Language Models Fit Brain Data
• Connecting Neural Response measurements & Computational Models of language: a non-comprehensive guide
• Brain-Like Language Processing via a Shallow Untrained Multihead Attention Network
• From Language to Cognition: How LLMs Outgrow the Human Language Network