Auto-Encoding Total Correlation Explanation

2018/02/16 by Shuyang Gao, Rob Brekelmans, Gao, Shuyang +5 · 12 citations
Computer Science · Mathematics · #Algorithms and Data Compression #Neural Networks and Applications #Time Series Analysis and Forecasting #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1802.05822

arxiv created 2018/02/16 · arxiv updated 2018/02/19

Abstract

Advances in unsupervised learning enable reconstruction and generation of samples from complex distributions, but this success is marred by the inscrutability of the representations learned. We propose an information-theoretic approach to characterizing disentanglement and dependence in representation learning using multivariate mutual information, also called total correlation. The principle of total Cor-relation Ex-planation (CorEx) has motivated successful unsupervised learning applications across a variety of domains, but under some restrictive assumptions. Here we relax those restrictions by introducing a flexible variational lower bound to CorEx. Surprisingly, we find that this lower bound is equivalent to the one in variational autoencoders (VAE) under certain conditions. This information-theoretic view of VAE deepens our understanding of hierarchical VAE and motivates a new algorithm, AnchorVAE, that makes latent codes more interpretable through information maximization and enables generation of richer and more realistic samples.

Citations

• Opening the Black Box of Deep Neural Networks via Information
• Independently Controllable Features
• Disentangling by Factorising

Cited by

• Putting An End to End-to-End: Gradient-Isolated Learning of Representations
• A Tutorial on Deep Latent Variable Models of Natural Language
• Improving Robustness and Generality of NLP Models Using Disentangled Representations
• IROTE: Human-like Traits Elicitation of Large Language Model via In-Context Self-Reflective Optimization
• PICACO: Pluralistic In-Context Value Alignment of LLMs via Total Correlation Optimization
• Structured Disentangled Representations
• Learning Disentangled Joint Continuous and Discrete Representations
• Multi-view Alignment and Generation in CCA via Consistent Latent Encoding
• Isolating Sources of Disentanglement in Variational Autoencoders
• InfoGAN-CR and ModelCentrality: Self-supervised Model Training and Selection for Disentangling GANs
• Dual Swap Disentangling
• Maximum Total Correlation Reinforcement Learning
• InfoCatVAE: Representation Learning with Categorical Variational Autoencoders

Related

• Isolating Sources of Disentanglement in Variational Autoencoders
• A Revisit of Total Correlation in Disentangled Variational Auto-Encoder with Partial Disentanglement
• Correlated Variational Auto-Encoders
• Unsupervised Learning via Total Correlation Explanation
• Learning Representations by Maximizing Mutual Information in Variational Autoencoders