3D View Prediction Models of the Dorsal Visual Stream

2023/09/04 by Sarch, Gabriel, Tung, Hsiao-Yu Fish, Wang, Aria +2
#Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Biological sciences #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neurons and Cognition (q-bio.NC)

paper · doi:10.48550/arxiv.2309.01782

Abstract

Deep neural network representations align well with brain activity in the ventral visual stream. However, the primate visual system has a distinct dorsal processing stream with different functional properties. To test if a model trained to perceive 3D scene geometry aligns better with neural responses in dorsal visual areas, we trained a self-supervised geometry-aware recurrent neural network (GRNN) to predict novel camera views using a 3D feature memory. We compared GRNN to self-supervised baseline models that have been shown to align well with ventral regions using the large-scale fMRI Natural Scenes Dataset (NSD). We found that while the baseline models accounted better for ventral brain regions, GRNN accounted for a greater proportion of variance in dorsal brain regions. Our findings demonstrate the potential for using task-relevant models to probe representational differences across visual streams.

Related

• Vision CNNs trained to estimate spatial latents learned similar ventral-stream-aligned representations
• Predictive coding video models capture dorsal parietal representations and human judgments for surfaces defined by motion
• STSBench: A Large-Scale Dataset for Modeling Neuronal Activity in the Dorsal Stream of Primate Visual Cortex
• A Dual-Stream Neural Network Explains the Functional Segregation of Dorsal and Ventral Visual Pathways in Human Brains
• End-to-end topographic networks as models of cortical map formation and human visual behaviour: moving beyond convolutions