Transparency of deep neural networks for medical image analysis: A review of interpretability methods

2021/12/04 by Zohaib Salahuddin, Henry C. Woodruff, Avishek Chatterjee +1 · 491 citations
Computer Science · Medicine · #Artificial Intelligence in Healthcare and Education #Artificial intelligence #Artificial neural network #Black box #Computer science #Computer security #Data science #Deep learning #Deep neural networks #Explainable Artificial Intelligence (XAI) #Interpretability #Machine learning #Radiomics and Machine Learning in Medical Imaging #Transparency (behavior) #Workflow

paper · pdf · doi:10.1016/j.compbiomed.2021.105111

published in Computers in Biology and Medicine 140, 105111 (Elsevier BV)

openalex publication_date 2021/12/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/04

Abstract

Artificial Intelligence (AI) has emerged as a useful aid in numerous clinical applications for diagnosis and treatment decisions. Deep neural networks have shown the same or better performance than clinicians in many tasks owing to the rapid increase in the available data and computational power. In order to conform to the principles of trustworthy AI, it is essential that the AI system be transparent, robust, fair, and ensure accountability. Current deep neural solutions are referred to as black-boxes due to a lack of understanding of the specifics concerning the decision-making process. Therefore, there is a need to ensure the interpretability of deep neural networks before they can be incorporated into the routine clinical workflow. In this narrative review, we utilized systematic keyword searches and domain expertise to identify nine different types of interpretability methods that have been used for understanding deep learning models for medical image analysis applications based on the type of generated explanations and technical similarities. Furthermore, we report the progress made towards evaluating the explanations produced by various interpretability methods. Finally, we discuss limitations, provide guidelines for using interpretability methods and future directions concerning the interpretability of deep neural networks for medical imaging analysis.

Citations

• ImageNet classification with deep convolutional neural networks

Cited by

• Deep learning interpretability in neuroimaging: A comprehensive survey and methodological recommendations
• COLLECT: A Counterfactual Oral Lesion Library for Trustworthy AI Evaluation
• Prototsnet: interpretable multivariate time series classification with prototypical parts
• SPROUT: A User-friendly, Scalable Toolkit for Multi-class Segmentation of Volumetric Images
• Imaging the hallmarks of cancer
• Adaptive Diagnostic Reasoning Framework for Pathology with Multimodal Large Language Models
• Care to Explain? AI Explanation Types Differentially Impact Chest Radiograph Diagnostic Performance and Physician Trust in AI
• Lung Nodule Classification Strategy via Dual‐Branch Capsule Network With Routing Optimization
• Evaluating the interpretability of clinical speech AI models: Lessons from two user studies
• Clinical-grade AI model for molecular subtyping of endometrial cancer: a multi-center cohort study in China
• A survey on neural-symbolic learning systems
• Bridging human and machine intelligence: Reverse-engineering radiologist intentions for clinical trust and adoption
• RAD: Towards Trustworthy Retrieval-Augmented Multi-modal Clinical Diagnosis
• Radiology Report Conditional 3D CT Generation with Multi Encoder Latent diffusion Model
• From Predictions to Explanations: Explainable AI for Autism Diagnosis and Identification of Critical Brain Regions
• FireGNN: Neuro-Symbolic Graph Neural Networks with Trainable Fuzzy Rules for Interpretable Medical Image Classification
• Medico 2025: Visual Question Answering for Gastrointestinal Imaging
• GazeLT: Visual attention-guided long-tailed disease classification in chest radiographs
• Interpretable weakly-supervised learning through kernel density matrices: A digital pathology use case
• An Efficient Approach for Muscle Segmentation and 3D Reconstruction Using Keypoint Tracking in MRI Scan
• Explainable Artificial Intelligence in Biomedical Image Analysis: A Comprehensive Survey
• PiPViT: Patch-based Visual Interpretable Prototypes for Retinal Image Analysis
• Explainable Anatomy-Guided AI for Prostate MRI: Foundation Models and In Silico Clinical Trials for Virtual Biopsy-based Risk Assessment
• Lung Nodule-SSM: Self-Supervised Lung Nodule Detection and Classification in Thoracic CT Images
• xEEGNet: Towards Explainable AI in EEG Dementia Classification
• MERA: Multimodal and Multiscale Self-Explanatory Model with Considerably Reduced Annotation for Lung Nodule Diagnosis
• A Design Framework for operationalizing Trustworthy Artificial Intelligence in Healthcare: Requirements, Tradeoffs and Challenges for its Clinical Adoption
• Learning to Harmonize Cross-vendor X-ray Images by Non-linear Image Dynamics Correction
• Diagnosis of Idiopathic Pulmonary Fibrosis in High-Resolution Computed Tomography Scans Using a Combination of Handcrafted Radiomics and Deep Learning. [europepmc]
• Current status and future developments in predicting outcomes in radiation oncology. [europepmc]
• An analysis-ready and quality controlled resource for pediatric brain white-matter research. [europepmc]
• Explainable medical imaging AI needs human-centered design: guidelines and evidence from a systematic review. [europepmc]
• Artificial Intelligence Methodologies Applied to Technologies for Screening, Diagnosis and Care of the Diabetic Foot: A Narrative Review. [europepmc]
• Criteria for the translation of radiomics into clinically useful tests. [europepmc]
• Survey of Explainable AI Techniques in Healthcare. [europepmc]
• Interpretable machine learning for dementia: A systematic review. [europepmc]
• Latent Similarity Identifies Important Functional Connections for Phenotype Prediction. [europepmc]
• On the Analyses of Medical Images Using Traditional Machine Learning Techniques and Convolutional Neural Networks. [europepmc]
• Improving disease classification performance and explainability of deep learning models in radiology with heatmap generators. [europepmc]
• Explainable deep learning approach for extracting cognitive features from hand-drawn images of intersecting pentagons. [europepmc]
• Revolutionizing Oral Cancer Detection: An Approach Using Aquila and Gorilla Algorithms Optimized Transfer Learning-Based CNNs. [europepmc]
• Medical image analysis using deep learning algorithms. [europepmc]
• Diagnostic performance of artificial intelligence-assisted PET imaging for Parkinson's disease: a systematic review and meta-analysis. [europepmc]
• Explainable ensemble learning method for OCT detection with transfer learning. [europepmc]
• Deep Learning in Breast Cancer Imaging: State of the Art and Recent Advancements in Early 2024. [europepmc]
• Enhancing brain tumor classification in MRI scans with a multi-layer customized convolutional neural network approach. [europepmc]
• Image-Based Artificial Intelligence in Psoriasis Assessment: The Beginning of a New Diagnostic Era? [europepmc]
• Care to Explain? AI Explanation Types Differentially Impact Chest Radiograph Diagnostic Performance and Physician Trust in AI. [europepmc]
• A literature review of artificial intelligence (AI) for medical image segmentation: from AI and explainable AI to trustworthy AI. [europepmc]
• The Role of Artificial Intelligence and Machine Learning Models in Antimicrobial Stewardship in Public Health: A Narrative Review. [europepmc]
• Explainable AI for lung cancer detection via a custom CNN on CT images. [europepmc]
• Explainable Artificial Intelligence in Radiological Cardiovascular Imaging-A Systematic Review. [europepmc]
• Application of artificial intelligence in medical imaging for tumor diagnosis and treatment: a comprehensive approach. [europepmc]

Related

• Transparency of Deep Neural Networks for Medical Image Analysis: A Review of Interpretability Methods
• Explainable deep learning models in medical image analysis
• On Interpretability of Artificial Neural Networks: A Survey
• Improving Interpretability of Deep Neural Networks in Medical Diagnosis by Investigating the Individual Units
• Deeply Explainable Artificial Neural Network