Multimodal neural networks better explain multivoxel patterns in the hippocampus

2021/12/11 by Choksi, Bhavin, Mozafari, Milad, VanRullen, Rufin +1
#FOS: Biological sciences #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural and Evolutionary Computing (cs.NE) #Neurons and Cognition (q-bio.NC)

paper · doi:10.48550/arxiv.2201.11517

Abstract

The human hippocampus possesses "concept cells", neurons that fire when presented with stimuli belonging to a specific concept, regardless of the modality. Recently, similar concept cells were discovered in a multimodal network called CLIP (Radford et at., 2021). Here, we ask whether CLIP can explain the fMRI activity of the human hippocampus better than a purely visual (or linguistic) model. We extend our analysis to a range of publicly available uni- and multi-modal models. We demonstrate that "multimodality" stands out as a key component when assessing the ability of a network to explain the multivoxel activity in the hippocampus.

Related

• An integrative view of human hippocampal function: Differences with other species and capacity considerations
• Uncovering Brain-Like Hierarchical Patterns in Vision-Language Models through fMRI-Based Neural Encoding
• Multi-modal brain encoding models for multi-modal stimuli
• Semantic Tuning of Single Neurons in the Human Medial Temporal Lobe
• Multimodal foundation models are better simulators of the human brain