Modeling short visual events through the BOLD moments video fMRI dataset and metadata

2024/07/24 by Benjamin Lahner, Kshitij Dwivedi, Polina Iamshchinina +10 · 1 voice · 2 citations
Computer Science · Neuroscience · #Neural dynamics and brain function #Visual Attention and Saliency Detection #Visual perception and processing mechanisms

paper · pdf · doi:10.1038/s41467-024-50310-3

openalex publication_date 2024/07/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

Studying the neural basis of human dynamic visual perception requires extensive experimental data to evaluate the large swathes of functionally diverse brain neural networks driven by perceiving visual events. Here, we introduce the BOLD Moments Dataset (BMD), a repository of whole-brain fMRI responses to over 1000 short (3 s) naturalistic video clips of visual events across ten human subjects. We use the videos' extensive metadata to show how the brain represents word- and sentence-level descriptions of visual events and identify correlates of video memorability scores extending into the parietal cortex. Furthermore, we reveal a match in hierarchical processing between cortical regions of interest and video-computable deep neural networks, and we showcase that BMD successfully captures temporal dynamics of visual events at second resolution. With its rich metadata, BMD offers new perspectives and accelerates research on the human brain basis of visual event perception.

Citations

• The mirror neuron system and action recognition
• Software tools for analysis and visualization of fMRI data
• Statistical parametric maps in functional imaging: A general linear approach
• N4ITK: Improved N3 Bias Correction
• ImageNet Large Scale Visual Recognition Challenge
• How Does the Brain Solve Visual Object Recognition?
• The Kinetics Human Action Video Dataset
• Neural correlates of action: Comparing meta-analyses of imagery, observation, and execution
• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million\n Narrated Video Clips
• Evidence for a Third Visual Pathway Specialized for Social Perception
• A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence
• A Penny for Your (visual) Thoughts: Self-Supervised Reconstruction of Natural Movies from Brain Activity
• A highly selective response to food in human visual cortex revealed by hypothesis-free voxel decomposition
• Visual representations in the human brain are aligned with large language models
• GIT: A Generative Image-to-text Transformer for Vision and Language

Cited by

• A large-scale heterogeneous 3D magnetic resonance brain imaging dataset for self-supervised learning
• NEvo: Neural-Guided Evolutionary Video Synthesis for Dynamic Visual Selectivity

Discussions

• Also from Amsterdam (Christina Sartzetaki, Gemma Roig, Cees G.M. Snoek, Iris Groen): Using RSA, 100 models are evaluated on the bold moments dataset (doi.org/10.1038/s414..., data available on openneu [bsky, 0 points, 1 comments]