Variational Information Bottleneck on Vector Quantized Autoencoders

2018/08/02 by Hanwei Wu, Wu, Hanwei, Markus Flierl +1 · 1 citation
Computer Science · Mathematics · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1808.01048

arxiv created 2018/08/02 · arxiv updated 2018/08/06

Abstract

In this paper, we provide an information-theoretic interpretation of the Vector Quantized-Variational Autoencoder (VQ-VAE). We show that the loss function of the original VQ-VAE can be derived from the variational deterministic information bottleneck (VDIB) principle. On the other hand, the VQ-VAE trained by the Expectation Maximization (EM) algorithm can be viewed as an approximation to the variational information bottleneck(VIB) principle.

Cited by

• Deconstructing Generative Diversity: An Information Bottleneck Analysis of Discrete Latent Generative Models

Related

• Information-theoretic Generalization Analysis for VQ-VAEs: A Role of Latent Variables
• Bounded Information Rate Variational Autoencoders
• Variational Encoders and Autoencoders : Information-theoretic Inference and Closed-form Solutions
• Deep Variational Information Bottleneck
• Training-Free Vector Quantization via Gaussian VAEs