Harmonizing the object recognition strategies of deep neural networks with humans

2022/11/08 by Thomas Fel, Fel, Thomas, Ivan Felipe +7 · 7 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Visual Attention and Saliency Detection #cs.AI #cs.CV

paper · pdf · doi:10.48550/arxiv.2211.04533

Published at NeurIPS 2022

openalex publication_date 2022/11/08 · arxiv created 2022/11/11 · arxiv updated 2022/11/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The many successes of deep neural networks (DNNs) over the past decade have largely been driven by computational scale rather than insights from biological intelligence. Here, we explore if these trends have also carried concomitant improvements in explaining the visual strategies humans rely on for object recognition. We do this by comparing two related but distinct properties of visual strategies in humans and DNNs: where they believe important visual features are in images and how they use those features to categorize objects. Across 84 different DNNs trained on ImageNet and three independent datasets measuring the where and the how of human visual strategies for object recognition on those images, we find a systematic trade-off between DNN categorization accuracy and alignment with human visual strategies for object recognition. State-of-the-art DNNs are progressively becoming less aligned with humans as their accuracy improves. We rectify this growing issue with our neural harmonizer: a general-purpose training routine that both aligns DNN and human visual strategies and improves categorization accuracy. Our work represents the first demonstration that the scaling laws that are guiding the design of DNNs today have also produced worse models of human vision. We release our code and data at https://serre-lab.github.io/Harmonization to help the field build more human-like DNNs.

Cited by

• Uncovering the Computational Ingredients of Human-Like Representations in LLMs
• AIM: Amending Inherent Interpretability via Self-Supervised Masking
• Why Uncertainty Calibration Matters for Reliable Perturbation-based Explanations
• Stretching Beyond the Obvious: A Gradient-Free Framework to Unveil the Hidden Landscape of Visual Invariance
• Quantifying task-relevant representational similarity using decision variable correlation
• CHART-6: Human-Centered Evaluation of Data Visualization Understanding in Vision-Language Models
• Enhancing Vision Transformer Explainability Using Artificial Astrocytes
• Better artificial intelligence does not mean better models of biology
• Improving Clinical Imaging Systems using Cognition based Approaches

Related

• Comparing deep neural networks against humans: object recognition when the signal gets weaker
• Performance-optimized deep neural networks are evolving into worse models of inferotemporal visual cortex
• Are Deep Neural Networks Adequate Behavioural Models of Human Visual Perception?
• Seeing eye-to-eye? A comparison of object recognition performance in humans and deep convolutional neural networks under image manipulation
• Improved object recognition using neural networks trained to mimic the brain's statistical properties