2018/12/17 by Mandana Hamidi‐Haines, Hamidi-Haines, Mandana, Zhongang Qi +7 · 1 citation
Biochemistry, Genetics and Molecular Biology · Computer Science · #Adversarial Robustness in Machine Learning #Cell Image Analysis Techniques #Computer Vision and Pattern Recognition (cs.CV) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Statistical and Computational Modeling
paper · pdf · doi:10.48550/arxiv.1812.07150
openalex publication_date 2018/12/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
We consider the problem of explaining the decisions of deep neural networks\nfor image recognition in terms of human-recognizable visual concepts. In\nparticular, given a test set of images, we aim to explain each classification\nin terms of a small number of image regions, or activation maps, which have\nbeen associated with semantic concepts by a human annotator. This allows for\ngenerating summary views of the typical reasons for classifications, which can\nhelp build trust in a classifier and/or identify example types for which the\nclassifier may not be trusted. For this purpose, we developed a user interface\nfor "interactive naming," which allows a human annotator to manually cluster\nsignificant activation maps in a test set into meaningful groups called "visual\nconcepts". The main contribution of this paper is a systematic study of the\nvisual concepts produced by five human annotators using the interactive naming\ninterface. In particular, we consider the adequacy of the concepts for\nexplaining the classification of test-set images, correspondence of the\nconcepts to activations of individual neurons, and the inter-annotator\nagreement of visual concepts. We find that a large fraction of the activation\nmaps have recognizable visual concepts, and that there is significant agreement\nbetween the different annotators about their denotations. Our work is an\nexploratory study of the interplay between machine learning and human\nrecognition mediated by visualizations of the results of learning.\n