Amortized Bethe Free Energy Minimization for Learning MRFs

2019/06/14 by Sam Wiseman, Yoon Kim, Wiseman, Sam +1
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Machine Learning and Data Classification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1906.06399

openalex publication_date 2019/06/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose to learn deep undirected graphical models (i.e., MRFs) with a non-ELBO objective for which we can calculate exact gradients. In particular, we optimize a saddle-point objective deriving from the Bethe free energy approximation to the partition function. Unlike much recent work in approximate inference, the derived objective requires no sampling, and can be efficiently computed even for very expressive MRFs. We furthermore amortize this optimization with trained inference networks. Experimentally, we find that the proposed approach compares favorably with loopy belief propagation, but is faster, and it allows for attaining better held out log likelihood than other recent approximate inference schemes.

Citations

• Annealed importance sampling
• Adam: A Method for Stochastic Optimization
• Belief Optimization for Binary Networks: A Stable Alternative to Loopy Belief Propagation
• Loopy Belief Propagation for Approximate Inference: An Empirical Study
• Auto-Encoding Variational Bayes
• Long Short-Term Memory
• Training Products of Experts by Minimizing Contrastive Divergence
• A tutorial on hidden Markov models and selected applications in speech recognition
• Gated Graph Sequence Neural Networks
• Pixel Recurrent Neural Networks
• Discriminative Embeddings of Latent Variable Models for Structured Data
• Semi-Supervised Classification with Graph Convolutional Networks
• Categorical Reparameterization with Gumbel-Softmax
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• REBAR: Low-variance, unbiased gradient estimates for discrete latent\n variable models
• A Tutorial on Deep Latent Variable Models of Natural Language
• A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
• Variational Inference with Normalizing Flows

Related

• Approximating the Bethe partition function
• Bethe Bounds and Approximating the Global Optimum
• Learning Deep Energy Models: Contrastive Divergence vs. Amortized MLE
• Pairwise MRF Calibration by Perturbation of the Bethe Reference Point
• Convexifying the Bethe Free Energy