2024/12/12 by Yu Xie, Yudi Xie, Xie, Yudi +11 · 3 voices · 1 citation
Biochemistry, Genetics and Molecular Biology · Computer Science · Earth and Planetary Sciences · Engineering · Environmental Science · #3D Surveying and Cultural Heritage #Artificial intelligence #Cartography #Computer science #Computer vision #Geography #Pattern recognition (psychology) #Remote Sensing and LiDAR Applications #Robotics and Sensor-Based Localization #cs.CV #cs.LG #cs.NE #q-bio.NC
paper · pdf · doi:10.48550/arxiv.2412.09115
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2024/12/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
Studies of the functional role of the primate ventral visual stream have traditionally focused on object categorization, often ignoring -- despite much prior evidence -- its role in estimating "spatial" latents such as object position and pose. Most leading ventral stream models are derived by optimizing networks for object categorization, which seems to imply that the ventral stream is also derived under such an objective. Here, we explore an alternative hypothesis: Might the ventral stream be optimized for estimating spatial latents? And a closely related question: How different -- if at all -- are representations learned from spatial latent estimation compared to categorization? To ask these questions, we leveraged synthetic image datasets generated by a 3D graphic engine and trained convolutional neural networks (CNNs) to estimate different combinations of spatial and category latents. We found that models trained to estimate just a few spatial latents achieve neural alignment scores comparable to those trained on hundreds of categories, and the spatial latent performance of models strongly correlates with their neural alignment. Spatial latent and category-trained models have very similar -- but not identical -- internal representations, especially in their early and middle layers. We provide evidence that this convergence is partly driven by non-target latent variability in the training data, which facilitates the implicit learning of representations of those non-target latents. Taken together, these results suggest that many training objectives, such as spatial latents, can lead to similar models aligned neurally with the ventral stream. Thus, one should not assume that the ventral stream is optimized for object categorization only. As a field, we need to continue to sharpen our measures of comparing models to brains to better understand the functional roles of the ventral stream.