2021/09/13 by Li Fuxin, Fuxin Li, Zhongang Qi +5
Computer Science · #Advanced Neural Network Applications #Artificial intelligence #Artificial neural network #Classifier (UML) #Computer science #Deep learning #Deep neural networks #Explainable Artificial Intelligence (XAI) #Image (mathematics) #Machine Learning and Data Classification #Machine learning #Process (computing) #Set (abstract data type) #Task (project management) #Visualization #cs.CV #cs.LG
paper · pdf · doi:10.1002/ail2.46
published as Applied AI Letters.2021;2:e46 · Submitted to Applied AI Letters
arxiv created 2021/09/13 · openalex publication_date 2021/11/06 · openalex created_date 2021/11/22 · arxiv updated 2022/03/14 · openalex updated_date 2026/08/01
This paper summarizes our endeavors in the past few years in terms of explaining image classifiers, with the aim of including negative results and insights we have gained. The paper starts with describing the explainable neural network (XNN), which attempts to extract and visualize several high-level concepts purely from the deep network, without relying on human linguistic concepts. This helps users understand network classifications that are less intuitive and substantially improves user performance on a difficult fine-grained classification task of discriminating among different species of seagulls. Realizing that an important missing piece is a reliable heatmap visualization tool, we have developed I-GOS and iGOS++ utilizing integrated gradients to avoid local optima in heatmap generation, which improved the performance across all resolutions. During the development of those visualizations, we realized that for a significant number of images, the classifier has multiple different paths to reach a confident prediction. This has lead to our recent development of structured attention graphs (SAGs), an approach that utilizes beam search to locate multiple coarse heatmaps for a single image, and compactly visualizes a set of heatmaps by capturing how different combinations of image regions impact the confidence of a classifier. Through the research process, we have learned much about insights in building deep network explanations, the existence and frequency of multiple explanations, and various tricks of the trade that make explanations work. In this paper, we attempt to share those insights and opinions with the readers with the hope that some of them will be informative for future researchers on explainable deep learning.