Do Language Models Know Who Did What to Whom?

2025/04/23 by Joseph M. Denning, Xiaohan Hannah Guo, Denning, Joseph M. +7 · 4 voices
Computer Science · Medicine · Neuroscience · Psychology · #Action Observation and Synchronization #Artificial Intelligence in Healthcare and Education #Cognition #Contrast (vision) #Focus (optics) #Information structure #Language model #Natural Language Processing Techniques #Neurobiology of Language and Bilingualism #Sentence #Similarity (geometry) #Thematic analysis #Thematic map #Thematic structure #Topic Modeling

paper · pdf · open access · doi:10.1162/opmi.a.365

published in Open Mind 10, 923-950 (The MIT Press)

openalex publication_date 2026/01/01 · openalex created_date 2026/07/17 · openalex updated_date 2026/07/25

Abstract

Language models (LMs) are commonly criticized for not "understanding" language. However, many critiques focus on cognitive abilities that, in humans, are distinct from language processing. Here, we instead study a kind of understanding tightly linked to language: inferring "who did what to whom" (thematic roles) in a sentence. Does the central training objective of LMs-word prediction-result in sentence representations that capture thematic roles? In two experiments, we characterized sentence representations in four LMs that have been proposed as models of human language processing. The overall representational similarity of sentence pairs did not reflect whether they had the same agent/patient assignments or opposite agent/patient assignments. Furthermore, we found limited evidence that thematic role information was available in any subspace of hidden activations. However, some attention heads robustly captured thematic roles, independently of syntax. Therefore, LMs can extract thematic roles but this information influences their representations weakly.

Citations

• Representation biases: will we achieve complete understanding by analyzing representations?
• How linguistics learned to stop worrying and love the language models
• Auxiliary task demands mask the capabilities of smaller language models
• The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets
• Language Models Represent Space and Time
• Prompting is not a substitute for probability measurements in large language models
• Training language models to follow instructions with human feedback
• Low-Dimensional Structure in the Space of Language Representations is\n Reflected in Brain Responses
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Discussions

• Do Large Language Models know who did what to whom? [hn, 39 points, 5 comments]
• Does training LLMs on word prediction result in human-like representation of thematic roles? Work by @josephdenning.bsky.social, Hannah Guo, and Bryor Snefjella who is no longer with us [bsky, 5 points, 2 comments]
• Do Large Language Models know who did what to whom? https://arxiv.org/abs/2504.16884 (https://news.ycombinator.com/item?id=43813879) [bsky, 0 points, 0 comments]
• Do Large Language Models know who did what to whom? https://arxiv.org/abs/2504.16884 [bsky, 0 points, 0 comments]

Related

• A Psycholinguistic Evaluation of Language Models' Sensitivity to Argument Roles
• Language Models as Agent Models
• What Can Language Models Tell Us About Human Cognition?
• Revealing emergent human-like conceptual representations from language prediction
• Do Language Models Learn Position-Role Mappings?