A Probabilistic Representation of Deep Learning for Improving The Information Theoretic Interpretability

2020/10/27 by Xinjie Lan, Lan, Xinjie, Kenneth E. Barner +1
Computer Science · Mathematics · #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Information Theory (cs.IT) #Machine Learning (cs.LG) #Machine Learning and Data Classification #Neural Networks and Applications #cs.IT #cs.LG #math.IT

paper · pdf · doi:10.48550/arxiv.2010.14054

arxiv created 2020/10/27 · openalex publication_date 2020/10/27 · arxiv updated 2020/10/28 · openalex created_date 2020/11/09 · openalex updated_date 2026/07/28

Abstract

In this paper, we propose a probabilistic representation of MultiLayer Perceptrons (MLPs) to improve the information-theoretic interpretability. Above all, we demonstrate that the activations being i.i.d. is not valid for all the hidden layers of MLPs, thus the existing mutual information estimators based on non-parametric inference methods, e.g., empirical distributions and Kernel Density Estimate (KDE), are invalid for measuring the information flow in MLPs. Moreover, we introduce explicit probabilistic explanations for MLPs: (i) we define the probability space (Omega_F, t, P_F) for a fully connected layer f and demonstrate the great effect of an activation function on the probability measure P_F ; (ii) we prove the entire architecture of MLPs as a Gibbs distribution P; and (iii) the back-propagation aims to optimize the sample space Omega_F of all the fully connected layers of MLPs for learning an optimal Gibbs distribution P* to express the statistical connection between the input and the label. Based on the probabilistic explanations for MLPs, we improve the information-theoretic interpretability of MLPs in three aspects: (i) the random variable of f is discrete and the corresponding entropy is finite; (ii) the information bottleneck theory cannot correctly explain the information flow in MLPs if we take into account the back-propagation; and (iii) we propose novel information-theoretic explanations for the generalization of MLPs. Finally, we demonstrate the proposed probabilistic representation and information-theoretic explanations for MLPs in a synthetic dataset and benchmark datasets.

Citations

• Adam: A Method for Stochastic Optimization
• An exact mapping between the Variational Renormalization Group and Deep\n Learning
• Gradient-based learning applied to document recognition
• Training Products of Experts by Minimizing Contrastive Divergence
• Opening the Black Box of Deep Neural Networks via Information
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes

Related

• A Probabilistic Representation of Deep Learning
• An Information-Theoretic Framework for Supervised Learning
• Information-Theoretic Framework for Understanding Modern Machine-Learning
• Information Lattice Learning as Probabilistic Graphical Model Structure Learning
• PAC-Bayesian Generalization Bounds for MultiLayer Perceptrons