Understanding Convolutional Neural Networks with Information Theory: An Initial Exploration

2018/04/18 by Shujian Yu, Kristoffer Wickstrøm, Yu, Shujian +5 · 2 citations
Computer Science · Physics and Astronomy · #FOS: Computer and information sciences #Image and Signal Denoising Methods #Information Theory (cs.IT) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications #Statistical Mechanics and Entropy

paper · pdf · doi:10.48550/arxiv.1804.06537

openalex publication_date 2018/04/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The matrix-based Renyi's α-entropy functional and its multivariate extension were recently developed in terms of the normalized eigenspectrum of a Hermitian matrix of the projected data in a reproducing kernel Hilbert space (RKHS). However, the utility and possible applications of these new estimators are rather new and mostly unknown to practitioners. In this paper, we first show that our estimators enable straightforward measurement of information flow in realistic convolutional neural networks (CNN) without any approximation. Then, we introduce the partial information decomposition (PID) framework and develop three quantities to analyze the synergy and redundancy in convolutional layer representations. Our results validate two fundamental data processing inequalities and reveal some fundamental properties concerning the training of CNN.

Citations

• The information bottleneck method
• Gradient-based learning applied to document recognition
• Estimation of Entropy and Mutual Information
• ImageNet classification with deep convolutional neural networks
• Evading the curse of dimensionality in nonparametric density estimation with simplified vine copulas
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Opening the Black Box of Deep Neural Networks via Information
• Efficient Estimation of Mutual Information for Strongly Dependent Variables
• Learning Multiple Layers of Features from Tiny Images

Cited by

• A Brain-like Synergistic Core in LLMs Drives Behaviour and Learning
• Information-Theoretic Greedy Layer-wise Training for Traffic Sign Recognition

Related

• Information Plane Analysis of Deep Neural Networks via Matrix-Based Renyi's Entropy and Tensor Kernels
• Understanding the Generalization Ability of Deep Learning Algorithms: A Kernelized Renyi's Entropy Perspective
• Neural Entropy
• Information Bottleneck Theory on Convolutional Neural Networks
• Exploring Information-Theoretic Metrics Associated with Neural Collapse in Supervised Training