A Survey on Interpretability in Visual Recognition

2025/07/15 by Wan, Qiyang, Gao, Chengzhi, Wang, Ruiping +1
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2507.11099

Abstract

In recent years, visual recognition methods have advanced significantly, finding applications across diverse fields. While researchers seek to understand the mechanisms behind the success of these models, there is also a growing impetus to deploy them in critical areas like autonomous driving and medical diagnostics to better diagnose failures, which promotes the development of interpretability research. This paper systematically reviews existing research on the interpretability of visual recognition models and proposes a taxonomy of methods from a human-centered perspective. The proposed taxonomy categorizes interpretable recognition methods based on Intent, Object, Presentation, and Methodology, thereby establishing a systematic and coherent set of grouping criteria for these XAI methods. Additionally, we summarize the requirements for evaluation metrics and explore new opportunities enabled by recent technologies, such as large multimodal models. We aim to organize existing research in this domain and inspire future investigations into the interpretability of visual recognition models.

Citations

• Unifying VXAI: A Systematic Review and Framework for the Evaluation of Explainable AI
• Building Trustworthy Multimodal AI: A Review of Fairness, Transparency, and Ethics in Vision-Language Tasks
• CoE: Chain-of-Explanation via Automatic Visual Concept Circuit Description and Polysemanticity Quantification
• Explainability for Vision Foundation Models: A Survey
• Finer-CAM: Spotting the Difference Reveals Finer Details for Visual Explanation
• A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Future
• Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey
• Interleaved-Modal Chain-of-Thought
• LLaVA-CoT: Let Vision Language Models Reason Step-by-Step
• Towards Interpreting Visual Information Processing in Vision-Language Models
• Self-eXplainable AI for Medical Image Analysis: A Survey and New Outlooks
• Navigating the Maze of Explainable AI: A Systematic Approach to Evaluating Methods and Metrics
• Blocks as Probes: Dissecting Categorization Ability of Large Multimodal Models
• Interpretable Clustering: A Survey
• GLEAMS: Bridging the Gap Between Local and Global Explanations
• Explain via Any Concept: Concept Bottleneck Model with Open Vocabulary Concepts
• Contrastive Learning with Counterfactual Explanations for Radiology Report Generation
• VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance
• Relevant Irrelevance: Generating Alterfactual Explanations for Image Classifiers
• Improving Concept Alignment in Vision-Language Concept Bottleneck Models
• Explainable AI (XAI) in Image Segmentation in Medicine, Industry, and Beyond: A Survey
• Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agenda
• Incremental Residual Concept Bottleneck Models
• Diffusion-based Iterative Counterfactual Explanations for Fetal Ultrasound Image Quality Assessment
• Vision Transformers with Natural Language Semantics
• Cross-Modal Projection in Multimodal LLMs Doesn't Really Project Visual Attributes to Textual Space
• MM-LLMs: Recent Advances in MultiModal Large Language Models
• Concept-based Explainable Artificial Intelligence: A Survey
• Understanding the (Extra-)Ordinary: Validating Deep Model Decisions with Prototypical Concept-based Explanations
• Multimodal Large Language Models: A Survey
• On the Relationship Between Interpretability and Explainability in Machine Learning
• Auxiliary Losses for Learning Generalizable Concept-based Models
• This Looks Like Those: Illuminating Prototypical Concepts Using Multiple Visualizations
• Latent Diffusion Counterfactual Explanations
• Explainable Artificial Intelligence for Drug Discovery and Development -- A Comprehensive Survey
• A Novel Neural-symbolic System under Statistical Relational Learning
• Interpretability-Aware Vision Transformer
• Distance-Aware eXplanation Based Learning
• DeViL: Decoding Vision features into Language
• Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence
• The Co-12 Recipe for Evaluating Interpretable Part-Prototype Image Classifiers
• Mitigating Bias: Enhancing Image Classification by Improving Model Explanations
• Adversarial attacks and defenses in explainable artificial intelligence: A survey
• Concept-Centric Transformers: Enhancing Model Interpretability through Object-Centric Concept Learning within a Shared Global Workspace
• Explain Any Concept: Segment Anything Meets Concept-Based Explanation
• Learning Bottleneck Concepts in Image Classification
• Label-Free Concept Bottleneck Models
• Micrograph segmentations for DDEVD
• Segment Anything
• HarsanyiNet: Computing Accurate Shapley Values in a Single Forward Propagation
• UFO: A unified method for controlling Understandability and Faithfulness Objectives in concept-based explanations for CNNs
• Zero-shot Model Diagnosis
• Reveal to Revise: An Explainable AI Life Cycle for Iterative Bias Correction of Deep Models
• Finding the right XAI method -- A Guide for the Evaluation and Ranking of Explainable AI Methods in Climate Science
• Learning Support and Trivial Prototypes for Interpretable Image Classification
• Optimizing Explanations by Network Canonization and Hyperparameter Search
• Attribution-based XAI Methods in Computer Vision: A Review
• Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification
• Learn to explain yourself, when you can: Equipping Concept Bottleneck Models with the ability to abstain on their concept predictions
• Diffusion Visual Counterfactual Explanations
• A Survey on Explainable Anomaly Detection
• Quantifying the Knowledge in a DNN to Explain Knowledge Distillation for Classification
• ELUDE: Generating interpretable explanations via a decomposition into labelled and unlabelled features
• ECLAD: Extracting Concepts with Local Aggregated Descriptors
• Xplique: A Deep Learning Explainability Toolbox
• From attribution maps to human-understandable explanations through Concept Relevance Propagation
• Explainable Artificial Intelligence (XAI) for Internet of Things: A Survey
• Explain to Not Forget: Defending Against Catastrophic Forgetting with XAI
• SegDiscover: Visual Concept Discovery via Unsupervised Semantic Segmentation
• Cycle-Consistent Counterfactuals by Latent Transformations
• Beyond Explaining: Opportunities and Challenges of XAI-Based Model Improvement
• Concept Bottleneck Model with Additional Unsupervised Concepts
• Natural Language Descriptions of Deep Visual Features
• From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI
• Interpretable Image Classification with Differentiable Prototypes Assignment
• Explainable Deep Learning in Healthcare: A Methodological Survey from an Attribution View
• Explainable AI (XAI): A Systematic Meta-Survey of Current Challenges and Future Opportunities
• Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities
• Visualizing the Emergence of Intermediate Visual Patterns in DNNs
• Counterfactual Explanation of Brain Activity Classifiers using Image-to-Image Transfer by Generative Adversarial Network
• A Framework for Learning Ante-hoc Explainable Models via Concepts
• Shared Interest: Measuring Human-AI Alignment to Identify Recurring Patterns in Model Behavior
• Interpretable Compositional Convolutional Neural Networks
• Synthetic Benchmarks for Scientific Research in Explainable Machine Learning
• A Game-Theoretic Taxonomy of Visual Concepts in DNNs
• Best of both worlds: local and global explanations with human-understandable concepts
• Keep CALM and Improve Visual Feature Attribution
• DISSECT: Disentangled Simultaneous Explanations via Concept Traversals
• IAIA-BL: A Case-based Interpretable Deep Learning Model for Classification of Mass Lesions in Digital Mammography
• Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges
• Learning Transferable Visual Models From Natural Language Supervision
• Conditional Generative Models for Counterfactual Explanations
• Neural Prototype Trees for Interpretable Fine-grained Image Recognition
• Use HiResCAM instead of Grad-CAM for faithful explanations of convolutional neural networks
• Captum: A unified and generic model interpretability library for PyTorch
• Attribute Prototype Network for Zero-Shot Learning
• Axiom-based Grad-CAM: Towards Accurate Visualization and Explanation of CNNs
• Concept Bottleneck Models
• Invertible Concept-based Explanations for CNN Models with Non-negative Concept Activation Vectors
• Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey
• InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs
• Explainable Deep Learning: A Field Guide for the Uninitiated
• xCos: An Explainable Cosine Metric for Face Verification Task
• Explaining Knowledge Distillation by Quantifying the Knowledge
• Sanity Checks for Saliency Metrics
• Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI
• On Completeness-aware Concept-Based Explanations in Deep Neural Networks
• Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks
• Towards a Unified Evaluation of Explanation Methods without Ground Truth
• Benchmarking Attribution Methods with Relative Feature Importance
• A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI
• Explaining Classifiers with Causal Concept Effect (CaCE)
• Interpretable Image Recognition with Hierarchical Prototypes
• Towards Aggregating Weighted Feature Attributions
• A Universal Logic Operator for Interpretable Deep Convolution Networks
• Deep Features Analysis with Attention Networks
• Interpretable CNNs for Object Classification
• Explainable and Explicit Visual Reasoning over Scene Graphs
• RISE: Randomized Input Sampling for Explanation of Black-box Models
• Revisiting the Importance of Individual Units in CNNs via Ablation
• Explaining Explanations: An Overview of Interpretability of Machine\n Learning
• Unsupervised Learning of Neural Networks to Explain Neural Networks
• Towards Interpretable Face Recognition
• Multimodal Explanations: Justifying Decisions and Pointing to the\n Evidence
• Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)
• Interpreting Deep Visual Representations via Network Dissection
• Deep Learning for Case-Based Reasoning through Prototypes: A Neural Network that Explains Its Predictions
• Interpretable Convolutional Neural Networks
• Towards Interpretable Deep Neural Networks by Leveraging Adversarial Examples
• Methods for Interpreting and Understanding Deep Neural Networks
• SmoothGrad: removing noise by adding noise
• Network Dissection: Quantifying Interpretability of Deep Visual\n Representations
• Axiomatic Attribution for Deep Networks
• Growing Interpretable Part Graphs on ConvNets via Multi-Shot Learning
• Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
• Top-down Neural Attention by Excitation Backprop
• Generating Visual Explanations
• Learning Deep Features for Discriminative Localization
• Evaluating the visualization of what a Deep Neural Network has learned
• LVLM-Interpret: An Interpretability Tool for Large Vision-Language Models
• Saliency strikes back: How filtering out high frequencies improves white-box explanations
• Gaze-Informed Vision Transformers: Predicting Driving Decisions Under Uncertainty
• Saliency-Bench: A Comprehensive Benchmark for Evaluating Visual Explanations

Related

• Visual Probing and Correction of Object Recognition Models with\n Interactive user feedback
• Interpretability of Machine Learning: Recent Advances and Future Prospects
• A Multimodal Automated Interpretability Agent
• The Definitions of Interpretability and Learning of Interpretable Models
• Multimodal LLM Augmented Reasoning for Interpretable Visual Perception Analysis