A Theoretical Explanation for Perplexing Behaviors of Backpropagation-based Visualizations

2018/05/17 by Weili Nie, Yang Zhang, Nie, Weili +3 · 6 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · #Artificial Intelligence (cs.AI) #Cell Image Analysis Techniques #Computer Vision and Pattern Recognition (cs.CV) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #cs.AI #cs.CV

paper · pdf · doi:10.48550/arxiv.1805.07039

21 pages, ICML 2018 (We revised the proofs of Theorem 1 and 2 in Appendix)

openalex publication_date 2018/05/17 · arxiv created 2020/02/13 · arxiv updated 2020/02/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Backpropagation-based visualizations have been proposed to interpret convolutional neural networks (CNNs), however a theory is missing to justify their behaviors: Guided backpropagation (GBP) and deconvolutional network (DeconvNet) generate more human-interpretable but less class-sensitive visualizations than saliency map. Motivated by this, we develop a theoretical explanation revealing that GBP and DeconvNet are essentially doing (partial) image recovery which is unrelated to the network decisions. Specifically, our analysis shows that the backward ReLU introduced by GBP and DeconvNet, and the local connections in CNNs are the two main causes of compelling visualizations. Extensive experiments are provided that support the theoretical analysis.

Citations

• Intriguing properties of neural networks
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps

Cited by

• The Directed Prediction Change - Efficient and Trustworthy Fidelity Assessment for Local Feature Attribution Methods
• Restricted Receptive Fields for Face Verification
• Attribution Explanations for Deep Neural Networks: A Theoretical Perspective
• Explaining Deep Learning Representations by Tracing the Training Process
• Hide-and-Seek: A Template for Explainable AI
• Definitions, methods, and applications in interpretable machine learning

Related

• How convolutional neural network see the world - A survey of convolutional neural network visualization methods
• How You See Me
• Deep learning for pedestrians: backpropagation in CNNs
• Understanding learned CNN features through Filter Decoding with Substitution
• Visualizing and Comparing Convolutional Neural Networks