Interpreting and Boosting Dropout from a Game-Theoretic View

2020/09/24 by Hao Zhang, Zhang, Hao, Sen Li +9 · 3 citations
Computer Science · Mathematics · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics #cs.AI #cs.CV #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2009.11729

openalex publication_date 2020/09/24 · openalex created_date 2020/10/01 · arxiv created 2021/03/16 · arxiv updated 2021/03/17 · openalex updated_date 2026/08/04

Abstract

This paper aims to understand and improve the utility of the dropout operation from the perspective of game-theoretic interactions. We prove that dropout can suppress the strength of interactions between input variables of deep neural networks (DNNs). The theoretic proof is also verified by various experiments. Furthermore, we find that such interactions were strongly related to the over-fitting problem in deep learning. Thus, the utility of dropout can be regarded as decreasing interactions to alleviate the significance of over-fitting. Based on this understanding, we propose an interaction loss to further improve the utility of dropout. Experimental results have shown that the interaction loss can effectively improve the utility of dropout and boost the performance of DNNs.

Citations

• Improving neural networks by preventing co-adaptation of feature detectors
• Gradient-based learning applied to document recognition
• Dropout Training as Adaptive Regularization
• ImageNet classification with deep convolutional neural networks
• Axiomatic Attribution for Deep Networks
• Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs
• Consistent Individualized Feature Attribution for Tree Ensembles
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Feature Interaction Interpretability: A Case for Explaining Ad-Recommendation Systems via Neural Interaction Detection
• Learning Multiple Layers of Features from Tiny Images
• Technical Note: Game-Theoretic Interactions of Different Orders
• Interpreting Multivariate Shapley Interactions in DNNs

Cited by

• The Interaction Bottleneck of Deep Neural Networks: Discovery, Proof, and Modulation
• Game-Theoretic Gradient Control for Robust Neural Network Training
• Technical Report: Quantifying and Analyzing the Generalization Power of a DNN

Related

• Dropout as a Regularizer of Interaction Effects
• On the Implicit Bias of Dropout
• Dropout as a Bayesian Approximation: Appendix
• Surprising properties of dropout in deep networks
• Understanding Dropout as an Optimization Trick