Auto-Encoding Total Correlation Explanation

2018/02/16 by Shuyang Gao, Rob Brekelmans, Gao, Shuyang +5 · 6 citations
Computer Science · #Time Series Analysis and Forecasting #Neural Networks and Applications #Algorithms and Data Compression

paper · pdf · doi:10.48550/arxiv.1802.05822

Abstract

Advances in unsupervised learning enable reconstruction and generation of samples from complex distributions, but this success is marred by the inscrutability of the representations learned. We propose an information-theoretic approach to characterizing disentanglement and dependence in representation learning using multivariate mutual information, also called total correlation. The principle of total Cor-relation Ex-planation (CorEx) has motivated successful unsupervised learning applications across a variety of domains, but under some restrictive assumptions. Here we relax those restrictions by introducing a flexible variational lower bound to CorEx. Surprisingly, we find that this lower bound is equivalent to the one in variational autoencoders (VAE) under certain conditions. This information-theoretic view of VAE deepens our understanding of hierarchical VAE and motivates a new algorithm, AnchorVAE, that makes latent codes more interpretable through information maximization and enables generation of richer and more realistic samples.

Citations

• Opening the Black Box of Deep Neural Networks via Information
• Independently Controllable Features
• Disentangling by Factorising

Cited by

• Putting An End to End-to-End: Gradient-Isolated Learning of Representations
• A Tutorial on Deep Latent Variable Models of Natural Language
• Improving Robustness and Generality of NLP Models Using Disentangled Representations
• IROTE: Human-like Traits Elicitation of Large Language Model via In-Context Self-Reflective Optimization
• Structured Disentangled Representations
• Learning Disentangled Joint Continuous and Discrete Representations

Related

• Isolating Sources of Disentanglement in Variational Autoencoders
• A Revisit of Total Correlation in Disentangled Variational Auto-Encoder with Partial Disentanglement
• Correlated Variational Auto-Encoders
• Learning Representations by Maximizing Mutual Information in Variational\n Autoencoders
• Disentanglement and Generalization Under Correlation Shifts