2015/06/12 by Jorg Bornschein, Jörg Bornschein, Samira Shabanian +6 · 8 citations
Computer Science · Mathematics · #Algorithm #Artificial intelligence #Bayesian inference #Bayesian probability #Bhattacharyya distance #Computer science #FOS: Computer and information sciences #Fiducial inference #Frequentist inference #Gaussian Processes and Bayesian Inference #Generative Adversarial Networks and Image Synthesis #Generative grammar #Generative model #Helmholtz free energy #Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine learning #Music and Audio Processing #cs.LG #stat.ML
paper · pdf · doi:10.48550/arxiv.1506.03877
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2015/06/12 · arxiv created 2016/05/25 · arxiv updated 2016/05/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Efficient unsupervised training and inference in deep generative models remains a challenging problem. One basic approach, called Helmholtz machine, involves training a top-down directed generative model together with a bottom-up auxiliary model used for approximate inference. Recent results indicate that better generative models can be obtained with better approximate inference procedures. Instead of improving the inference procedure, we here propose a new model which guarantees that the top-down and bottom-up distributions can efficiently invert each other. We achieve this by interpreting both the top-down and the bottom-up directed models as approximate inference distributions and by defining the model distribution to be the geometric mean of these two. We present a lower-bound for the likelihood of this model and we show that optimizing this bound regularizes the model so that the Bhattacharyya distance between the bottom-up and top-down approximate distributions is minimized. This approach results in state of the art generative models which prefer significantly deeper architectures while it allows for orders of magnitude more efficient approximate inference.