An Information Theoretic Interpretation to Deep Neural Networks

2019/05/16 by Shao‐Lun Huang, Xiangxiang Xu, Huang, Shao-Lun +5 · 3 citations
Computer Science · #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Information Theory (cs.IT) #Machine Learning (cs.LG) #Machine Learning and Algorithms #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.1905.06600

openalex publication_date 2019/05/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

It is commonly believed that the hidden layers of deep neural networks (DNNs) attempt to extract informative features for learning tasks. In this paper, we formalize this intuition by showing that the features extracted by DNN coincide with the result of an optimization problem, which we call the `universal feature selection' problem, in a local analysis regime. We interpret the weights training in DNN as the projection of feature functions between feature spaces, specified by the network structure. Our formulation has direct operational meaning in terms of the performance for inference tasks, and gives interpretations to the internal computation results of DNNs. Results of numerical experiments are provided to support the analysis.

Cited by

• Emerging AI Approaches for Cancer Spatial Omics
• Demystifying Spectral Feature Learning for Instrumental Variable Regression
• Information Science Principles of Machine Learning: A Causal Chain Meta-Framework Based on Formalized Information Mapping

Related

• Where is the Information in a Deep Neural Network?
• Opening the Black Box of Deep Neural Networks via Information
• Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values
• Information Flow in Deep Neural Networks
• A Local Geometric Interpretation of Feature Extraction in Deep Feedforward Neural Networks