Activation Matching for Explanation Generation

2025/09/27 by Suhail, Pirzada, Anand, Aditya, Sethi, Amit · 1 citation
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2509.23051

Abstract

In this paper we introduce an activation-matching--based approach to generate minimal, faithful explanations for the decision-making of a pretrained classifier on any given image. Given an input image x and a frozen model f, we train a lightweight autoencoder to output a binary mask m such that the explanation e = m \odot x preserves both the model's prediction and the intermediate activations of \(x\). Our objective combines: (i) multi-layer activation matching with KL divergence to align distributions and cross-entropy to retain the top-1 label for both the image and the explanation; (ii) mask priors -- L1 area for minimality, a binarization penalty for crisp 0/1 masks, and total variation for compactness; and (iii) abductive constraints for faithfulness and necessity. Together, these objectives yield small, human-interpretable masks that retain classifier behavior while discarding irrelevant input regions, providing practical and faithful minimalist explanations for the decision making of the underlying model.

Citations

• A Comprehensive Guide to Explainable AI: From Classical Models to LLMs
• Concept-Based Explanations in Computer Vision: Where Are We and Where Could We Go?
• Network Inversion of Convolutional Neural Nets
• Network Inversion of Binarised Neural Nets
• Leveraging Explanations in Interactive Machine Learning: An Overview
• Landscape Learning for Neural Network Inversion
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Abduction-Based Explanations for Machine Learning Models
• Explaining Explanations: An Overview of Interpretability of Machine\n Learning
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
• Synthesizing the preferred inputs for neurons in neural networks via\n deep generator networks
• Deep Residual Learning for Image Recognition
• Understanding Neural Networks Through Deep Visualization
• Understanding Deep Image Representations by Inverting Them

Cited by

• EXP-CAM: Explanation Generation and Circuit Discovery Using Classifier Activation Matching

Related

• EXP-CAM: Explanation Generation and Circuit Discovery Using Classifier Activation Matching
• Activation-Deactivation: A General Framework for Robust Post-hoc Explainable AI
• Model Interpretability and Rationale Extraction by Input Mask Optimization
• Towards Better Explanations of Class Activation Mapping
• P-TAME: Explain Any Image Classifier with Trained Perturbations