Investigating Saturation Effects in Integrated Gradients

2020/10/23 by Vivek Miglani, Narine Kokhlikyan, Miglani, Vivek +7 · 1 citation
Computer Science · #Adversarial Robustness in Machine Learning #Computer Vision and Pattern Recognition (cs.CV) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning in Healthcare

paper · pdf · doi:10.48550/arxiv.2010.12697

openalex publication_date 2020/10/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Integrated Gradients has become a popular method for post-hoc model interpretability. De-spite its popularity, the composition and relative impact of different regions of the integral path are not well understood. We explore these effects and find that gradients in saturated regions of this path, where model output changes minimally, contribute disproportionately to the computed attribution. We propose a variant of IntegratedGradients which primarily captures gradients in unsaturated regions and evaluate this method on ImageNet classification networks. We find that this attribution technique shows higher model faithfulness and lower sensitivity to noise com-pared with standard Integrated Gradients. A note-book illustrating our computations and results is available at https://github.com/vivekmig/captum-1/tree/ExpandedIG.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Gradients of Counterfactuals
• Axiomatic Attribution for Deep Networks
• RISE: Randomized Input Sampling for Explanation of Black-box Models
• Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
• On a Method to Measure Supervised Multiclass Model’s Interpretability: Application to Degradation Diagnosis (Short Paper)
• Generalized Integrated Gradients: A practical method for explaining diverse ensembles

Cited by

• Probing Deep into Temporal Profile Makes the Infrared Small Target Detector Much Better

Related

• Integrated Decision Gradients: Compute Your Attributions Where the Model Makes Its Decision
• Using the Path of Least Resistance to Explain Deep Networks
• A Rigorous Study of Integrated Gradients Method and Extensions to Internal Neuron Attributions
• Geometrically Guided Integrated Gradients
• A New Baseline Assumption of Integated Gradients Based on Shaply value