Visualizing Information Bottleneck through Variational Inference

2022/12/24 by Cipta Herwana, Herwana, Cipta, Abhishek Kadian +1
Computer Science · #Stochastic Gradient Optimization Techniques #Adversarial Robustness in Machine Learning #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2212.12667

Abstract

The Information Bottleneck theory provides a theoretical and computational framework for finding approximate minimum sufficient statistics. Analysis of the Stochastic Gradient Descent (SGD) training of a neural network on a toy problem has shown the existence of two phases, fitting and compression. In this work, we analyze the SGD training process of a Deep Neural Network on MNIST classification and confirm the existence of two phases of SGD training. We also propose a setup for estimating the mutual information for a Deep Neural Network through Variational Inference.

Related

• Drill the Cork of Information Bottleneck by Inputting the Most Important Data
• Learning Robust Variational Information Bottleneck with Reference
• Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression
• Information Bottleneck and its Applications in Deep Learning
• Information Flow in Deep Neural Networks