2021/01/06 by Linh Tran, Maja Pantić, Maja Pantic +4 · 1 citation
Computer Science · Mathematics · #Artificial intelligence #Artificial neural network #Autoencoder #Bayesian Methods and Mixture Models #Bayesian probability #Cauchy distribution #Cluster analysis #Computer science #Constraint (computer-aided design) #Divergence (linguistics) #Face recognition and analysis #Gaussian #Generative Adversarial Networks and Image Synthesis #Inference #Kullback–Leibler divergence #Latent variable #Machine learning #Mathematics #Mixture model #Pattern recognition (psychology) #Prior probability #Statistics #cs.CV #cs.LG
paper · pdf · doi:10.48550/arxiv.2101.02149
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2021/01/06 · arxiv created 2021/02/12 · arxiv updated 2021/02/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06
Recent work in unsupervised learning has focused on efficient inference and learning in latent variables models. Training these models by maximizing the evidence (marginal likelihood) is typically intractable. Thus, a common approximation is to maximize the Evidence Lower BOund (ELBO) instead. Variational autoencoders (VAE) are a powerful and widely-used class of generative models that optimize the ELBO efficiently for large datasets. However, the VAE's default Gaussian choice for the prior imposes a strong constraint on its ability to represent the true posterior, thereby degrading overall performance. A Gaussian mixture model (GMM) would be a richer prior, but cannot be handled efficiently within the VAE framework because of the intractability of the Kullback-Leibler divergence for GMMs. We deviate from the common VAE framework in favor of one with an analytical solution for Gaussian mixture prior. To perform efficient inference for GMM priors, we introduce a new constrained objective based on the Cauchy-Schwarz divergence, which can be computed analytically for GMMs. This new objective allows us to incorporate richer, multi-modal priors into the autoencoding framework. We provide empirical studies on a range of datasets and show that our objective improves upon variational auto-encoding models in density estimation, unsupervised clustering, semi-supervised learning, and face analysis.