2024/09/03 by Maytus Piriyajitakonkij, Sirawaj Itthipuripat, Piriyajitakonkij, Maytus +5
Computer Science · Engineering · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Biological sciences #FOS: Computer and information sciences #Face recognition and analysis #Image Processing Techniques and Applications #Neurons and Cognition (q-bio.NC) #Visual Attention and Saliency Detection
paper · pdf · doi:10.48550/arxiv.2409.02241
openalex publication_date 2024/09/03 · openalex created_date 2024/10/23 · openalex updated_date 2026/07/28
In visual decision making, high-level features, such as object categories, have a strong influence on choice. However, the impact of low-level features on behavior is less understood partly due to the high correlation between high- and low-level features in the stimuli presented (e.g., objects of the same category are more likely to share low-level features). To disentangle these effects, we propose a method that de-correlates low- and high-level visual properties in a novel set of stimuli. Our method uses two Convolutional Neural Networks (CNNs) as candidate models of the ventral visual stream: the CORnet-S that has high neural predictivity in high-level, IT-like responses and the VGG-16 that has high neural predictivity in low-level responses. Triplets (root, image1, image2) of stimuli are parametrized by the level of low- and high-level similarity of images extracted from the different layers. These stimuli are then used in a decision-making task where participants are tasked to choose the most similar-to-the-root image. We found that different networks show differing abilities to predict the effects of low-versus-high-level similarity: while CORnet-S outperforms VGG-16 in explaining human choices based on high-level similarity, VGG-16 outperforms CORnet-S in explaining human choices based on low-level similarity. Using Brain-Score, we observed that the behavioral prediction abilities of different layers of these networks qualitatively corresponded to their ability to explain neural activity at different levels of the visual hierarchy. In summary, our algorithm for stimulus set generation enables the study of how different representations in the visual stream affect high-level cognitive behaviors.