2025/04/01 by Anna Bavaresco, Raquel Fernández, Bavaresco, Anna +1 · 2 citations
Computer Science · Neuroscience · Psychology · #Action Observation and Synchronization #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Neurobiology of Language and Bilingualism
paper · pdf · doi:10.48550/arxiv.2504.00942
openalex publication_date 2025/04/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
A common assumption in Computational Linguistics is that text representations learnt by multimodal models are richer and more human-like than those by language-only models, as they are grounded in images or audio -- similar to how human language is grounded in real-world experiences. However, empirical studies checking whether this is true are largely lacking. We address this gap by comparing word representations from contrastive multimodal models vs. language-only ones in the extent to which they capture experiential information -- as defined by an existing norm-based 'experiential model' -- and align with human fMRI responses. Our results indicate that, surprisingly, language-only models are superior to multimodal ones in both respects. Additionally, they learn more unique brain-relevant semantic information beyond that shared with the experiential model. Overall, our study highlights the need to develop computational models that better integrate the complementary semantic information provided by multimodal data sources.