Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse

2019/11/06 by James Lucas, George Tucker, Lucas, James +5 · 10 citations
Computer Science · Physics and Astronomy · #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Image and Signal Denoising Methods #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Model Reduction and Neural Networks

paper · pdf · doi:10.48550/arxiv.1911.02469

openalex publication_date 2019/11/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Posterior collapse in Variational Autoencoders (VAEs) arises when the variational posterior distribution closely matches the prior for a subset of latent variables. This paper presents a simple and intuitive explanation for posterior collapse through the analysis of linear VAEs and their direct correspondence with Probabilistic PCA (pPCA). We explain how posterior collapse may occur in pPCA due to local maxima in the log marginal likelihood. Unexpectedly, we prove that the ELBO objective for the linear VAE does not introduce additional spurious local maxima relative to log marginal likelihood. We show further that training a linear VAE with exact variational inference recovers an identifiable global maximum corresponding to the principal component directions. Empirically, we find that our linear analysis is predictive even for high-capacity, non-linear VAEs and helps explain the relationship between the observation noise, local maxima, and posterior collapse in deep Gaussian VAEs.

Citations

• Adam: A Method for Stochastic Optimization
• Robust principal component analysis?
• VAE with a VampPrior
• Tackling Over-pruning in Variational Autoencoders
• Inference Suboptimality in Variational Autoencoders
• Semi-Amortized Variational Autoencoders
• Avoiding Latent Variable Collapse With Generative Skip Models
• Lagging Inference Networks and Posterior Collapse in Variational Autoencoders
• Diagnosing and Enhancing VAE Models
• BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling

Cited by

• Spatiotemporal Satellite Image Downscaling with Transfer Encoders and Autoregressive Generative Models
• Self-Supervised Compression and Artifact Correction for Streaming Underwater Imaging Sonar
• Deep Jump Gaussian Processes for Surrogate Modeling of High-Dimensional Piecewise Continuous Functions
• Neural Variational Dropout Processes
• Posterior Collapse as a Phase Transition in Variational Autoencoders
• Harnessing Synthetic Data from Generative AI for Statistical Inference
• Inverse Design of Amorphous Materials with Targeted Properties
• Few-step Flow for 3D Generation via Marginal-Data Transport Distillation
• Toward Architecture-Agnostic Local Control of Posterior Collapse in VAEs
• Zero-Variance Gradients for Variational Autoencoders

Related

• Beyond Vanilla Variational Autoencoders: Detecting Posterior Collapse in Conditional and Hierarchical Variational Autoencoders
• Posterior Collapse as a Phase Transition in Variational Autoencoders
• High-dimensional Asymptotics of VAEs: Threshold of Posterior Collapse and Dataset-Size Dependence of Rate-Distortion Curve
• The Usual Suspects? Reassessing Blame for VAE Posterior Collapse
• Lagging Inference Networks and Posterior Collapse in Variational Autoencoders