Deep Variational Information Bottleneck

2016/12/31 by Alexander A. Alemi, Ian Fischer, Joshua V. Dillon +1 · 15 citations
Computer Science · Mathematics · #cs.LG #cs.IT #math.IT

paper · pdf

published as Proceedings of the International Conference on Learning Representations (ICLR) 2017 · 19 pages, 8 figures, Accepted to ICLR17

arxiv created 2019/10/23 · arxiv updated 2019/10/25

Abstract

We present a variational approximation to the information bottleneck of Tishby et al. (1999). This variational approach allows us to parameterize the information bottleneck model using a neural network and leverage the reparameterization trick for efficient training. We call this method "Deep Variational Information Bottleneck", or Deep VIB. We show that models trained with the VIB objective outperform those that are trained with other forms of regularization, in terms of generalization performance and robustness to adversarial attack.

Cited by

• SynLeaF: A Dual-Stage Multimodal Fusion Framework for Synthetic Lethality Prediction Across Pan- and Single-Cancer Contexts
• On the Rate-Distortion-Complexity Tradeoff for Semantic Communication
• Learning to Perceive the World Through Control: Empowerment-Based Representation Learning
• Source-Side Sufficiency for the Information Bottleneck: Exact Reduction and Finite-Block Equivalence
• A Unifying Framework for Unsupervised Concept Extraction
• Generalization Guarantees for Multi-View Representation Learning and Application to Regularization via Gaussian Product Mixture Prior
• HoRFFI: High-Openness RF Fingerprint Identification with a Similarity-Enhanced Variational Information Bottleneck
• UG-UMRE: Uncertainty-Guided Modality Augmentation and Distributional Calibration for Unified Multimodal Relation Extraction
• SJEPA: Learning Elegant Latent Dynamics with Hybrid Symbolic-Neural Predictors
• Learning to Score
• Towards Explainable Fusion and Balanced Learning in Multimodal Sentiment Analysis
• SkeletonX: Data-Efficient Skeleton-based Action Recognition via Cross-sample Feature Aggregation
• Scalable Multi-task Edge Sensing via Task-oriented Joint Information Gathering and Broadcast
• Generalized Audio Deepfake Detection Using Frame-level Latent Information Entropy
• Transformer-based audio-visual multimodal fusion for fine-grained recognition of individual sow nursing behaviour

Related

• Learning Robust Variational Information Bottleneck with Reference
• Tighter Bounds on the Information Bottleneck with Application to Deep Learning
• Visualizing Information Bottleneck through Variational Inference
• Flexible Variational Information Bottleneck: Achieving Diverse Compression with a Single Training
• The Variational InfoMax Learning Objective