Variational Inference: A Review for Statisticians
2023/01/01 by David M. Blei, Alp Kucukelbir, Jon McAuliffe
Computer Science · Mathematics · #Gaussian Processes and Bayesian Inference #Bayesian Methods and Mixture Models #Statistical Methods and Inference
paper · doi:10.6084/m9.figshare.5203696.v2
openalex publication_date 2023/01/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/22
Abstract
One of the core problems of modern statistics is to approximate difficult-to-compute probability densities. This problem is especially important in Bayesian statistics, which frames all inference about unknown quantities as a calculation involving the posterior density. In this article, we review variational inference (VI), a method from machine learning that approximates probability densities through optimization. VI has been used in many applications and tends to be faster than classical methods, such as Markov chain Monte Carlo sampling. The idea behind VI is to first posit a family of densities and then to find a member of that family which is close to the target density. Closeness is measured by Kullback–Leibler divergence. We review the ideas behind mean-field variational inference, discuss the special case of VI applied to exponential family models, present a full example with a Bayesian mixture of Gaussians, and derive a variant that uses stochastic optimization to scale up to massive data. We discuss modern research in VI and highlight important open problems. VI is powerful, but it is not yet well understood. Our hope in writing this article is to catalyze statistical research on this class of algorithms. Supplementary materials for this article are available online.
Cited by
- A Bayesian Fisher-EM algorithm for discriminative Gaussian subspace\n clustering
- Probabilistic Circuits for Variational Inference in Discrete Graphical Models
- A Nonparametric Multi-view Model for Estimating Cell Type-Specific Gene\n Regulatory Networks
- Monte Carlo Approximation of Bayes Factors via Mixing With Surrogate Distributions
- Visual Adversarial Imitation Learning using Variational Models
- GP-VAE: Deep Probabilistic Time Series Imputation
- Projected Stein Variational Newton: A Fast and Scalable Bayesian Inference Method in High Dimensions
- Scalable Bayesian Optimization with Sparse Gaussian Process Models
- Monte Carlo Co-Ordinate Ascent Variational Inference
- When Gaussian Process Meets Big Data: A Review of Scalable GPs
- The k-tied Normal Distribution: A Compact Parameterization of Gaussian\n Mean Field Posteriors in Bayesian Neural Networks
- Semi-Amortized Variational Autoencoders
- Leveraging Uncertainty for Improved Static Malware Detection Under Extreme False Positive Constraints
- Sinkhorn EM: An Expectation-Maximization algorithm based on entropic optimal transport
- Variational Marginal Particle Filters
- Improving Inference for Neural Image Compression
- Hindsight Expectation Maximization for Goal-conditioned Reinforcement Learning
- Bayesian Functional Principal Components Analysis via Variational\n Message Passing
- Supervised Uncertainty Quantification for Segmentation with Multiple Annotations
- Thompson Sampling for Dynamic Pricing
- Exploring Variational Deep Q Networks
- Static Analysis for Probabilistic Programs
- Bayesian Attention Modules
- Bayesian neural networks for flight trajectory prediction and safety assessment
- Sequential Variational Autoencoders for Collaborative Filtering
- Bayesian inference for network Poisson models
- A review of uncertainty quantification in deep learning: Techniques, applications and challenges
- Attentional Prototype Inference for Few-Shot Segmentation
- Cataloging the Visible Universe through Bayesian Inference at Petascale
- Deep Learning from Noisy Image Labels with Quality Embedding
- Advances in Variational Inference
- Machine learning in agricultural and applied economics
- Joint Mapping and Calibration via Differentiable Sensor Fusion
- Stein Variational Gradient Descent Without Gradient
- Evidential Turing Processes
- Practical bounds on the error of Bayesian posterior approximations: A nonasymptotic approach
- Variational Inference for Category Recommendation in E-Commerce platforms
- Epileptic Seizure Forecasting: Probabilistic seizure-risk assessment and data-fusion
- Variational Causal Networks: Approximate Bayesian Inference over Causal Structures
- Probabilistic Spatial Transformer Networks
- Bayesian Nonparametric Reinforcement Learning in LTE and Wi-Fi Coexistence
- Leveraging the Exact Likelihood of Deep Latent Variable Models
- Variational Bayesian Inference for Mixed Logit Models with Unobserved\n Inter- and Intra-Individual Heterogeneity
- Scalable Nonparametric Sampling from Multimodal Posteriors with the Posterior Bootstrap
- Scalable Bayesian neural networks by layer-wise input augmentation
- Object-Centric Image Generation with Factored Depths, Locations, and\n Appearances
- Empirical Bayes Matrix Factorization
- Variational Inference with Holder Bounds
- Uncertainty Quantification in Deep Learning for Safer Neuroimage Enhancement
- Variational Bayesian Monte Carlo with Noisy Likelihoods
- Variational Bayesian Decision-making for Continuous Utilities
- An Instability in Variational Inference for Topic Models
- Matched bipartite block model with covariates
- Adaptively stacking ensembles for influenza forecasting with incomplete data
- VIREL: A Variational Inference Framework for Reinforcement Learning
- Covariances, Robustness, and Variational Bayes
- Bayesian statistics and modelling
- Mathematics for Machine Learning
- Quadruply Stochastic Gaussian Processes
- Efficient Semi-Implicit Variational Inference
- Beyond the Mean-Field: Structured Deep Gaussian Processes Improve the\n Predictive Uncertainties
- Variational Langevin Hamiltonian Monte Carlo for Distant Multi-modal Sampling
- Variational inequalities and mean-field approximations for partially observed systems of queueing networks
- Semi-Implicit Graph Variational Auto-Encoders
- Bayesian Probabilistic Numerical Integration with Tree-Based Models
- All in the Exponential Family: Bregman Duality in Thermodynamic Variational Inference
- Game-Theoretic Multiagent Reinforcement Learning
- Model-Based Clustering of Nonparametric Weighted Networks With Application to Water Pollution Analysis
- Uncertainty Estimation in Deep Neural Networks for Point Cloud\n Segmentation in Factory Planning
- Pathfinder: Parallel quasi-Newton variational inference
- Convergence Rates of Variational Posterior Distributions
- A deep generative model for gene expression profiles from single-cell RNA sequencing
- Gradient Boosted Normalizing Flows
- Infinite mixtures of multivariate normal-inverse Gaussian distributions for clustering of skewed data
- System inference for the spatio-temporal evolution of infectious diseases: Michigan in the time of COVID-19
- A similarity-based Bayesian mixture-of-experts model
- Scalable Bayesian Inverse Reinforcement Learning
- Projected Stein Variational Gradient Descent
- Relating Graph Neural Networks to Structural Causal Models
- A Framework for Interdomain and Multioutput Gaussian Processes
- Strong consistency of the AIC, BIC, Cp and KOO methods in high-dimensional multivariate linear regression
- A Parsimonious Tour of Bayesian Model Uncertainty
- Semi-Implicit Variational Inference
- Spherical Sliced-Wasserstein
- Disentangling the Gauss-Newton Method and Approximate Inference for Neural Networks
- Stochastic Aggregation in Graph Neural Networks
- Video Anomaly Detection and Localization via Gaussian Mixture Fully Convolutional Variational Autoencoder
- Sequential Matrix Completion
- Equipping SBMs with RBMs: An Explainable Approach for Analysis of Networks with Covariates
- Deep Probabilistic Imaging: Uncertainty Quantification and Multi-modal Solution Characterization for Computational Imaging
- Traversing Time with Multi-Resolution Gaussian Process State-Space Models
- Augmented KRnet for density estimation and approximation
- Probabilistic Recurrent State-Space Models
- Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse\n Coding
- Variational inference for cutting feedback in misspecified models
- Stochastic Gradient MCMC with Repulsive Forces
- Estimating the Rate Constant from Biosensor Data via an Adaptive Variational Bayesian Approach
- Variational Bayesian Context-aware Representation for Grocery Recommendation
- Replicating Active Appearance Model by Generator Network
- Asymptotic Consistency of α-Rényi-Approximate Posteriors
- Understanding Uncertainty in Bayesian Deep Learning
- Balancing Reconstruction Quality and Regularisation in ELBO for VAEs
- A Tutorial on Deep Latent Variable Models of Natural Language
- Design by adaptive sampling
- Deep Recurrent Gaussian Process with Variational Sparse Spectrum Approximation
- Provable Smoothness Guarantees for Black-Box Variational Inference
- Joint Variational Autoencoders for Recommendation with Implicit Feedback
- Gaussian Process Bandit Optimization of the Thermodynamic Variational\n Objective
- Non-Parametric Calibration for Classification
- Loss function based second-order Jensen inequality and its application to particle variational inference
- Variational Inference for Shrinkage Priors: The R package vir
- Graphite: Iterative Generative Modeling of Graphs
- Variational Item Response Theory: Fast, Accurate, and Expressive
- Mixture Representation Learning with Coupled Autoencoders
- Approximation Based Variance Reduction for Reparameterization Gradients
- Desiderata for Representation Learning: A Causal Perspective
- Laplace-aided variational inference for differential equation models
- Adaptive and Calibrated Ensemble Learning with Dependent Tail-free Process
- Seq2Tens: An Efficient Representation of Sequences by Low-Rank Tensor Projections
- A Forest Mixture Bound for Block-Free Parallel Inference
- A Theoretical Case Study of Structured Variational Inference for Community Detection
- Manifold Optimization Assisted Gaussian Variational Approximation
- Towards Patient Record Summarization Through Joint Phenotype Learning in HIV Patients
- Nonparametric Deconvolution Models
- Markov-Modulated Hawkes Processes for Sporadic and Bursty Event Occurrences
- Distiller: A Systematic Study of Model Distillation Methods in Natural Language Processing
- Metric Gaussian Variational Inference
- Semi-Parametric Hierarchical Bayes Estimates of New Yorkers' Willingness\n to Pay for Features of Shared Automated Vehicle Services
- Variational Bayesian Unlearning
- Probabilistic Data Analysis with Probabilistic Programming
- Generative Models for Security: Attacks, Defenses, and Opportunities
- Quasi-symplectic Langevin Variational Autoencoder
- Contributions to Large Scale Bayesian Inference and Adversarial Machine Learning
- Variational Refinement for Importance Sampling Using the Forward Kullback-Leibler Divergence
- When random initializations help: a study of variational inference for community detection
- Scaling Bayesian inference of mixed multinomial logit models to very large datasets
- Whence the Expected Free Energy?
- Deep Active Inference as Variational Policy Gradients
- Low Complexity Approximate Bayesian Logistic Regression for Sparse Online Learning
- CWAE-IRL: Formulating a supervised approach to Inverse Reinforcement Learning problem
- Gaussian Process Inference Using Mini-batch Stochastic Gradient Descent: Convergence Guarantees and Empirical Benefits
- Variational inference formulation for a model-free simulation of a dynamical system with unknown parameters by a recurrent neural network
- Challenges and Opportunities in High-dimensional Variational Inference
- What You Say and How You Say it: Joint Modeling of Topics and Discourse in Microblog Conversations
- Implicit Generative Modeling for Efficient Exploration
- Measure Transport with Kernel Stein Discrepancy
- Learning to Learn Variational Semantic Memory
- Variational Bayes method for ordinary differential equation models
- Energy-Inspired Models: Learning with Sampler-Induced Distributions
- Versatile Inverse Reinforcement Learning via Cumulative Rewards
- Deep active inference agents using Monte-Carlo methods
- Online neural connectivity estimation with ensemble stimulation
- A Mathematical Walkthrough and Discussion of the Free Energy Principle
- Consistency of ELBO maximization for model selection
- CRAUM-Net: Contextual Recursive Attention with Uncertainty Modeling for Salient Object Detection
- Bandit Learning for Diversified Interactive Recommendation
- New Heuristics for Parallel and Scalable Bayesian Optimization
- Instance-dependent Label-noise Learning under a Structural Causal Model
- Convergence Rates of Empirical Bayes Posterior Distributions: A Variational Perspective
- Voice Conversion Based Speaker Normalization for Acoustic Unit Discovery
- Bayesian neural networks and dimensionality reduction
- Deep Bayesian Active Learning for Multiple Correct Outputs
- Rao-Blackwellized Stochastic Gradients for Discrete Distributions
- Robust, Accurate Stochastic Optimization for Variational Inference
- Modeling Graph Node Correlations with Neighbor Mixture Models
- Mean-Field Variational Inference for Gradient Matching with Gaussian Processes
- Mini-batch Metropolis-Hastings MCMC with Reversible SGLD Proposal
- Information-based Variational Model Reduction of high-dimensional Reaction Networks
- Adaptive Nonparametric Psychophysics
- Quasi-Newton Quasi-Monte Carlo for variational Bayes
- Angle-Based Models for Ranking Data
- Reward-rational (implicit) choice: A unifying formalism for reward learning
- Large-Scale Wasserstein Gradient Flows
- Regional Topics in British Grocery Retail Transactions
- Stacking for Non-mixing Bayesian Computations: The Curse and Blessing of Multimodal Posteriors
- Posterior Meta-Replay for Continual Learning
- Uncertainty-guided Continual Learning with Bayesian Neural Networks
- Neural Stochastic Differential Equations: Deep Latent Gaussian Models in\n the Diffusion Limit
- Multi-Task Variational Information Bottleneck
- Bayesian model selection in the \M-open setting --\n Approximate posterior inference and probability-proportional-to-size\n subsampling for efficient large-scale leave-one-out cross-validation
- Inferring Spatial Uncertainty in Object Detection
- Probabilistic Active Meta-Learning
- The Thermodynamic Variational Objective
- Variational Variance: Simple, Reliable, Calibrated Heteroscedastic Noise Variance Parameterization
- Inverse Gaussian Process regression for likelihood-free inference
- Towards Scale-Invariant Graph-related Problem Solving by Iterative Homogeneous Graph Neural Networks
- Wat zei je? Detecting Out-of-Distribution Translations with Variational Transformers
Related