Flow of Information in Feed-Forward Deep Neural Networks

2016/03/20 by Pejman Khadivi, Khadivi, Pejman, Ravi Tandon +3 · 2 citations
Computer Science · Mathematics · #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Information Theory (cs.IT) #Machine Learning (cs.LG) #Machine Learning and Algorithms #Neural Networks and Applications #cs.IT #cs.LG #math.IT

paper · pdf · doi:10.48550/arxiv.1603.06220

arxiv created 2016/03/20 · openalex publication_date 2016/03/20 · arxiv updated 2016/03/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Feed-forward deep neural networks have been used extensively in various machine learning applications. Developing a precise understanding of the underling behavior of neural networks is crucial for their efficient deployment. In this paper, we use an information theoretic approach to study the flow of information in a neural network and to determine how entropy of information changes between consecutive layers. Moreover, using the Information Bottleneck principle, we develop a constrained optimization problem that can be used in the training process of a deep neural network. Furthermore, we determine a lower bound for the level of data representation that can be achieved in a deep neural network with an acceptable level of distortion.

Citations

• The information bottleneck method
• An exact mapping between the Variational Renormalization Group and Deep Learning
• Deep Learning in Neural Networks: An Overview

Cited by

• Feedforward Ordering in Neural Connectomes via Feedback Arc Minimization
• Do Compressed Representations Generalize Better?

Related

• Information Flow in Deep Neural Networks
• Deep Learning and the Information Bottleneck Principle
• Understanding Autoencoders with Information Theoretic Concepts
• Opening the Black Box of Deep Neural Networks via Information
• Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression