2020/03/13 by Luke Nicholas Darlow, Amos Storkey, Darlow, Luke Nicholas +1
Computer Science · Mathematics · #Adversarial Robustness in Machine Learning #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Image and Signal Denoising Methods #Machine Learning (cs.LG) #Machine Learning (stat.ML) #cs.CV #cs.LG #stat.ML
paper · pdf · doi:10.48550/arxiv.2003.06254
10 pages + appendices; submitted to ICLR 2019
arxiv created 2020/03/13 · openalex publication_date 2020/03/13 · arxiv updated 2020/03/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
The information bottleneck principle (Shwartz-Ziv & Tishby, 2017) suggests that SGD-based training of deep neural networks results in optimally compressed hidden layers, from an information theoretic perspective. However, this claim was established on toy data. The goal of the work we present here is to test whether the information bottleneck principle is applicable to a realistic setting using a larger and deeper convolutional architecture, a ResNet model. We trained PixelCNN++ models as inverse representation decoders to measure the mutual information between hidden layers of a ResNet and input image data, when trained for (1) classification and (2) autoencoding. We find that two stages of learning happen for both training regimes, and that compression does occur, even for an autoencoder. Sampling images by conditioning on hidden layers' activations offers an intuitive visualisation to understand what a ResNets learns to forget.