Variational Inference: A Review for Statisticians
2023/01/01 by David M. Blei, Alp Kucukelbir, Jon McAuliffe
Computer Science · Mathematics · #Bayesian Methods and Mixture Models #Gaussian Processes and Bayesian Inference #Statistical Methods and Inference
paper · doi:10.6084/m9.figshare.5203696.v2
openalex publication_date 2023/01/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/22
Abstract
One of the core problems of modern statistics is to approximate difficult-to-compute probability densities. This problem is especially important in Bayesian statistics, which frames all inference about unknown quantities as a calculation involving the posterior density. In this article, we review variational inference (VI), a method from machine learning that approximates probability densities through optimization. VI has been used in many applications and tends to be faster than classical methods, such as Markov chain Monte Carlo sampling. The idea behind VI is to first posit a family of densities and then to find a member of that family which is close to the target density. Closeness is measured by Kullback–Leibler divergence. We review the ideas behind mean-field variational inference, discuss the special case of VI applied to exponential family models, present a full example with a Bayesian mixture of Gaussians, and derive a variant that uses stochastic optimization to scale up to massive data. We discuss modern research in VI and highlight important open problems. VI is powerful, but it is not yet well understood. Our hope in writing this article is to catalyze statistical research on this class of algorithms. Supplementary materials for this article are available online.
Citations
Cited by
- A Bayesian Fisher-EM algorithm for discriminative Gaussian subspace clustering
- Probabilistic Circuits for Variational Inference in Discrete Graphical Models
- A Nonparametric Multi-view Model for Estimating Cell Type-Specific Gene Regulatory Networks
- Monte Carlo Approximation of Bayes Factors via Mixing with Surrogate Distributions
- Visual Adversarial Imitation Learning using Variational Models
- GP-VAE: Deep Probabilistic Time Series Imputation
- Projected Stein Variational Newton: A Fast and Scalable Bayesian Inference Method in High Dimensions
- Scalable Bayesian Optimization with Sparse Gaussian Process Models
- Monte Carlo Co-Ordinate Ascent Variational Inference
- When Gaussian Process Meets Big Data: A Review of Scalable GPs
- The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks
- Semi-Amortized Variational Autoencoders
- Leveraging Uncertainty for Improved Static Malware Detection Under Extreme False Positive Constraints
- Sinkhorn EM: An Expectation-Maximization algorithm based on entropic optimal transport
- Variational Marginal Particle Filters
- Improving Inference for Neural Image Compression
- Hindsight Expectation Maximization for Goal-conditioned Reinforcement Learning
- Bayesian Functional Principal Components Analysis via Variational Message Passing
- Supervised Uncertainty Quantification for Segmentation with Multiple Annotations
- Thompson Sampling for Dynamic Pricing
- Exploring Variational Deep Q Networks
- Static Analysis for Probabilistic Programs
- Bayesian Attention Modules
- Bayesian neural networks for flight trajectory prediction and safety assessment
- Sequential Variational Autoencoders for Collaborative Filtering
- Bayesian inference for network Poisson models
- A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
- Attentional Prototype Inference for Few-Shot Segmentation
- Cataloging the Visible Universe through Bayesian Inference at Petascale
- Deep Learning from Noisy Image Labels with Quality Embedding
- Advances in Variational Inference
- Machine learning in agricultural and applied economics
- Joint Mapping and Calibration via Differentiable Sensor Fusion
- Stein Variational Gradient Descent Without Gradient
- Evidential Turing Processes
- Practical bounds on the error of Bayesian posterior approximations: A nonasymptotic approach
- Variational Inference for Category Recommendation in E-Commerce platforms
- Epileptic Seizure Forecasting: Probabilistic seizure-risk assessment and data-fusion
- Variational Causal Networks: Approximate Bayesian Inference over Causal Structures
- Probabilistic Spatial Transformer Networks
- Bayesian Nonparametric Reinforcement Learning in LTE and Wi-Fi Coexistence
- Leveraging the Exact Likelihood of Deep Latent Variable Models
- Variational Bayesian Inference for Mixed Logit Models with Unobserved Inter- and Intra-Individual Heterogeneity
- Scalable Nonparametric Sampling from Multimodal Posteriors with the Posterior Bootstrap
- Scalable Bayesian neural networks by layer-wise input augmentation
- Object-Centric Image Generation with Factored Depths, Locations, and\n Appearances
- Empirical Bayes Matrix Factorization
- Variational Inference with Holder Bounds
- Uncertainty Quantification in Deep Learning for Safer Neuroimage Enhancement
- Variational Bayesian Monte Carlo with Noisy Likelihoods
- Variational Bayesian Decision-making for Continuous Utilities
- An Instability in Variational Inference for Topic Models
- Matched bipartite block model with covariates
- Adaptively stacking ensembles for influenza forecasting with incomplete data
- VIREL: A Variational Inference Framework for Reinforcement Learning
- Covariances, Robustness, and Variational Bayes
- Bayesian statistics and modelling
- Mathematics for Machine Learning
- Quadruply Stochastic Gaussian Processes
- Efficient Semi-Implicit Variational Inference
- Beyond the Mean-Field: Structured Deep Gaussian Processes Improve the Predictive Uncertainties
- Variational Langevin Hamiltonian Monte Carlo for Distant Multi-modal Sampling
- Variational inequalities and mean-field approximations for partially observed systems of queueing networks
- Semi-Implicit Graph Variational Auto-Encoders
- Bayesian Probabilistic Numerical Integration with Tree-Based Models
- All in the Exponential Family: Bregman Duality in Thermodynamic Variational Inference
- Game-Theoretic Multiagent Reinforcement Learning
- Model-Based Clustering of Nonparametric Weighted Networks with Application to Water Pollution Analysis
- Uncertainty Estimation in Deep Neural Networks for Point Cloud Segmentation in Factory Planning
- Pathfinder: Parallel quasi-Newton variational inference
- Convergence Rates of Variational Posterior Distributions
- A deep generative model for gene expression profiles from single-cell RNA sequencing
- Gradient Boosted Normalizing Flows
- Infinite mixtures of multivariate normal-inverse Gaussian distributions for clustering of skewed data
- System inference for the spatio-temporal evolution of infectious diseases: Michigan in the time of COVID-19
- A similarity-based Bayesian mixture-of-experts model
- Scalable Bayesian Inverse Reinforcement Learning
- Projected Stein Variational Gradient Descent
- Relating Graph Neural Networks to Structural Causal Models
- A Framework for Interdomain and Multioutput Gaussian Processes
- Strong consistency of the AIC, BIC, Cp and KOO methods in high-dimensional multivariate linear regression
- A Parsimonious Tour of Bayesian Model Uncertainty
- Semi-Implicit Variational Inference
- Spherical Sliced-Wasserstein
- Disentangling the Gauss-Newton Method and Approximate Inference for Neural Networks
- Stochastic Aggregation in Graph Neural Networks
- Video Anomaly Detection and Localization via Gaussian Mixture Fully Convolutional Variational Autoencoder
- Sequential Matrix Completion
- Equipping SBMs with RBMs: An Explainable Approach for Analysis of Networks with Covariates
- Deep Probabilistic Imaging: Uncertainty Quantification and Multi-modal Solution Characterization for Computational Imaging
- Traversing Time with Multi-Resolution Gaussian Process State-Space Models
- Augmented KRnet for density estimation and approximation
- Probabilistic Recurrent State-Space Models
- Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding
- Variational inference for cutting feedback in misspecified models
- Stochastic Gradient MCMC with Repulsive Forces
- Estimating the Rate Constant from Biosensor Data via an Adaptive Variational Bayesian Approach
- Approximation Properties of Variational Bayes for Vector Autoregressions
- Variational Bayesian Context-aware Representation for Grocery Recommendation
- Replicating Active Appearance Model by Generator Network
- Asymptotic Consistency of α-Rényi-Approximate Posteriors
- Understanding Uncertainty in Bayesian Deep Learning
- Balancing Reconstruction Quality and Regularisation in ELBO for VAEs
- A Tutorial on Deep Latent Variable Models of Natural Language
- Design by adaptive sampling
- Deep Recurrent Gaussian Process with Variational Sparse Spectrum Approximation
- Provable Smoothness Guarantees for Black-Box Variational Inference
- Joint Variational Autoencoders for Recommendation with Implicit Feedback
- Gaussian Process Bandit Optimization of the Thermodynamic Variational Objective
- Non-Parametric Calibration for Classification
- Loss function based second-order Jensen inequality and its application to particle variational inference
- Variational Inference for Shrinkage Priors: The R package vir
- Graphite: Iterative Generative Modeling of Graphs
- Variational Item Response Theory: Fast, Accurate, and Expressive
- Mixture Representation Learning with Coupled Autoencoders
- Approximation Based Variance Reduction for Reparameterization Gradients
- Desiderata for Representation Learning: A Causal Perspective
- Laplace-aided variational inference for differential equation models
- Adaptive and Calibrated Ensemble Learning with Dependent Tail-free Process
- Seq2Tens: An Efficient Representation of Sequences by Low-Rank Tensor Projections
- A Forest Mixture Bound for Block-Free Parallel Inference
- A Theoretical Case Study of Structured Variational Inference for Community Detection
- Manifold Optimization Assisted Gaussian Variational Approximation
- Towards Patient Record Summarization Through Joint Phenotype Learning in HIV Patients
- Nonparametric Deconvolution Models
- Markov-Modulated Hawkes Processes for Sporadic and Bursty Event Occurrences
- Distiller: A Systematic Study of Model Distillation Methods in Natural Language Processing
- Metric Gaussian Variational Inference
- Semi-Parametric Hierarchical Bayes Estimates of New Yorkers' Willingness to Pay for Features of Shared Automated Vehicle Services
- Variational Bayesian Unlearning
- Probabilistic Data Analysis with Probabilistic Programming
- Generative Models for Security: Attacks, Defenses, and Opportunities
- Quasi-symplectic Langevin Variational Autoencoder
- Contributions to Large Scale Bayesian Inference and Adversarial Machine Learning
- Variational Refinement for Importance Sampling Using the Forward Kullback-Leibler Divergence
- When random initializations help: a study of variational inference for community detection
- Scaling Bayesian inference of mixed multinomial logit models to very large datasets
- Whence the Expected Free Energy?
- Deep Active Inference as Variational Policy Gradients
- Low Complexity Approximate Bayesian Logistic Regression for Sparse Online Learning
- CWAE-IRL: Formulating a supervised approach to Inverse Reinforcement Learning problem
- Gaussian Process Inference Using Mini-batch Stochastic Gradient Descent: Convergence Guarantees and Empirical Benefits
- Variational inference formulation for a model-free simulation of a dynamical system with unknown parameters by a recurrent neural network
- Challenges and Opportunities in High-dimensional Variational Inference
- What You Say and How You Say it: Joint Modeling of Topics and Discourse in Microblog Conversations
- Implicit Generative Modeling for Efficient Exploration
- Measure Transport with Kernel Stein Discrepancy
- Learning to Learn Variational Semantic Memory
- Variational Bayes method for ordinary differential equation models
- Energy-Inspired Models: Learning with Sampler-Induced Distributions
- Versatile Inverse Reinforcement Learning via Cumulative Rewards
- Deep active inference agents using Monte-Carlo methods
- Online neural connectivity estimation with ensemble stimulation
- A Mathematical Walkthrough and Discussion of the Free Energy Principle
- Consistency of ELBO maximization for model selection
- CRAUM-Net: Contextual Recursive Attention with Uncertainty Modeling for Salient Object Detection
- Bandit Learning for Diversified Interactive Recommendation
- New Heuristics for Parallel and Scalable Bayesian Optimization
- Instance-dependent Label-noise Learning under a Structural Causal Model
- Convergence Rates of Empirical Bayes Posterior Distributions: A Variational Perspective
- Voice Conversion Based Speaker Normalization for Acoustic Unit Discovery
- VAE-KRnet and its applications to variational Bayes
- Bayesian neural networks and dimensionality reduction
- Deep Bayesian Active Learning for Multiple Correct Outputs
- Rao-Blackwellized Stochastic Gradients for Discrete Distributions
- Robust, Accurate Stochastic Optimization for Variational Inference
- Modeling Graph Node Correlations with Neighbor Mixture Models
- Mean-Field Variational Inference for Gradient Matching with Gaussian Processes
- Mini-batch Metropolis-Hastings MCMC with Reversible SGLD Proposal
- Information-based Variational Model Reduction of high-dimensional Reaction Networks
- Adaptive Nonparametric Psychophysics
- Quasi-Newton Quasi-Monte Carlo for variational Bayes
- Angle-Based Models for Ranking Data
- Reward-rational (implicit) choice: A unifying formalism for reward learning
- Large-Scale Wasserstein Gradient Flows
- Regional Topics in British Grocery Retail Transactions
- Stacking for Non-mixing Bayesian Computations: The Curse and Blessing of Multimodal Posteriors
- Posterior Meta-Replay for Continual Learning
- Uncertainty-guided Continual Learning with Bayesian Neural Networks
- Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit
- Multi-Task Variational Information Bottleneck
- Bayesian model selection in the M-open setting -- Approximate posterior inference and probability-proportional-to-size subsampling for efficient large-scale leave-one-out cross-validation
- Inferring Spatial Uncertainty in Object Detection
- Probabilistic Active Meta-Learning
- The Thermodynamic Variational Objective
- Variational Variance: Simple, Reliable, Calibrated Heteroscedastic Noise Variance Parameterization
- Inverse Gaussian Process regression for likelihood-free inference
- Towards Scale-Invariant Graph-related Problem Solving by Iterative Homogeneous Graph Neural Networks
- Wat zei je? Detecting Out-of-Distribution Translations with Variational Transformers
- LeMoNADe: Learned Motif and Neuronal Assembly Detection in calcium imaging videos
- The Dynamic Embedded Topic Model
- Learning from a lot: Empirical Bayes in high-dimensional prediction settings
- A Free Lunch from the Noise: Provable and Practical Exploration for Representation Learning
- Assessment and adjustment of approximate inference algorithms using the law of total variance
- Topic-Aware Neural Keyphrase Generation for Social Media Language
- Clustering microbiome data using mixtures of logistic normal multinomial models
- Semi-Supervised Learning with the Deep Rendering Mixture Model
- Approximating exponential family models (not single distributions) with a two-network architecture
- Boosting Variational Inference
- Direct Optimization through arg max for Discrete Variational Auto-Encoder
- Prescribed Generative Adversarial Networks
- Variational Bayesian modelling of mixed-effects
- Nearly Optimal Variational Inference for High Dimensional Regression with Shrinkage Priors
- Natural Language Generation with Neural Variational Models
- Bayesian Meta-Learning Through Variational Gaussian Processes
- Deep Probabilistic Ensembles: Approximate Variational Inference through KL Regularization
- Using Social Network Information in Bayesian Truth Discovery
- Locally Learned Synaptic Dropout for Complete Bayesian Inference
- Expected path length on random manifolds
- Black-box density function estimation using recursive partitioning
- BAR: Bayesian Activity Recognition using variational inference
- Understanding Variational Inference in Function-Space
- Geometry of Friston's active inference
- Flexible mean field variational inference using mixtures of non-overlapping exponential families
- Synergetic Learning Systems: Concept, Architecture, and Algorithms
- Accelerated Flow for Probability Distributions
- Bayesian Inference Forgetting
- Variational Hetero-Encoder Randomized GANs for Joint Image-Text Modeling
- Uncertainty estimation under model misspecification in neural network regression
- LonelyText: A Short Messaging Based Classification of Loneliness
- Variational Deep Q Network
- Active Learning Solution on Distributed Edge Computing
- Unsupervised Contextual Anomaly Detection using Joint Deep Variational Generative Models
- Recommendation from Raw Data with Adaptive Compound Poisson Factorization
- Learning GPLVM with arbitrary kernels using the unscented transformation
- New Tricks for Estimating Gradients of Expectations
- Joint Target Detection and Tracking in Multipath Environment: A Variational Bayesian Approach
- Variational Bayesian Quantization
- Segment-Based Credit Scoring Using Latent Clusters in the Variational Autoencoder
- Convergence Rates of Variational Inference in Sparse Deep Learning
- Top-N Recommendation with Counterfactual User Preference Simulation
- Multilevel Monte Carlo estimation of log marginal likelihood
- A Dynamic Edge Exchangeable Model for Sparse Temporal Networks
- TAP free energy, spin glasses, and variational inference
- Data-driven discovery of interpretable causal relations for deep learning material laws with uncertainty propagation
- Modeling Item Response Theory with Stochastic Variational Inference
- Bounded Regression with Gaussian Process Projection
- Variational Inference in high-dimensional linear regression
- Automatic Guide Generation for Stan via NumPyro
- Bayesian Full-waveform Inversion with Realistic Priors
- Discrete Action On-Policy Learning with Action-Value Critic
- Probabilistic Software Modeling: A Data-driven Paradigm for Software Analysis
- Supervised multi-specialist topic model with applications on large-scale electronic health record data
- A Bayesian Perspective on Training Speed and Model Selection
- Time-Variant Variational Transfer for Value Functions
- A Simple Algorithm for Scalable Monte Carlo Inference
- A Bayesian Perspective of Convolutional Neural Networks through a Deconvolutional Generative Model
- A Statistical Introduction to Template Model Builder: A Flexible Tool for Spatial Modeling
- Posterior inference unchained with EL2O
- Text-Based Ideal Points
- Batch Selection for Parallelisation of Bayesian Quadrature
- PAC-Bayesian Generalization Bounds for MultiLayer Perceptrons
- Variational Autoencoders for Learning Nonlinear Dynamics of Physical Systems
- Pseudo-marginal Bayesian inference for supervised Gaussian process latent variable models
- Copula-like Variational Inference
- Learning an Astronomical Catalog of the Visible Universe through Scalable Bayesian Inference
- Spectrum Gaussian Processes Based On Tunable Basis Functions
- Nonnegative spatial factorization
- DrNAS: Dirichlet Neural Architecture Search
- A novel variational Bayesian method for variable selection in logistic regression models
- Fast and Scalable Spike and Slab Variable Selection in High-Dimensional Gaussian Processes
- Black-Box Inference for Non-Linear Latent Force Models
- Labels Are Not Perfect: Inferring Spatial Uncertainty in Object Detection
- A Class of Conjugate Priors for Multinomial Probit Models which Includes the Multivariate Normal One
- A Stochastic Quasi-Newton Method for Large-Scale Nonconvex Optimization with Applications
- KORELASI PANJANG TUNGKAI DAN DAYA LEDAK OTOTTUNGKAI TERHADAP JAUHNYA HASIL TENDANGANBOLA PADA SISWA EKSTRAKURIKULER SEPAK BOLASMA NEGERI 1 NGRAMBE KABUPATEN NGAWITAHUN 2008/2009
- Wasserstein Variational Inference
- Statistical modeling of rates and trends in Holocene relative sea level
- Information Criterion for Boltzmann Approximation Problems
- Nonlinear Hawkes Processes in Time-Varying System
- Testing whether a Learning Procedure is Calibrated
- Latent space projection predictive inference
- Scalable Modeling of Spatiotemporal Data using the Variational Autoencoder: an Application in Glaucoma
- Spectral Subsampling MCMC for Stationary Time Series
- Exact and approximate inference in graphical models: variable elimination and beyond
- A Bit More Bayesian: Domain-Invariant Learning with Uncertainty
- Adversarial Learning of Poisson Factorisation Model for Gauging Brand Sentiment in User Reviews
- Probabilistic Modeling for Novelty Detection with Applications to Fraud Identification
- Continuous-Time Model-Based Reinforcement Learning
- Identification, Interpretability, and Bayesian Word Embeddings
- Large deviations of empirical measures of diffusions in weighted topologies
- Leveraging Cross Feedback of User and Item Embeddings with Attention for Variational Autoencoder based Collaborative Filtering
- Concentrated Document Topic Model
- A semi-supervised deep learning algorithm for abnormal EEG identification
- Variational Inference with Numerical Derivatives: variance reduction through coupling
- On MCMC for variationally sparse Gaussian processes: A pseudo-marginal approach
- Structured Stochastic Gradient MCMC
- Information processing constraints in travel behaviour modelling: A generative learning approach
- Variational Encoders and Autoencoders : Information-theoretic Inference and Closed-form Solutions
- Information Theoretic Structured Generative Modeling
- Cloud2Curve: Generation and Vectorization of Parametric Sketches
- Bayesian Hierarchical Modeling: Application Towards Production Results in the Eagle Ford Shale of South Texas
- A Bayesian Feature Allocation Model for Identification of Cell Subpopulations Using Cytometry Data
- Bayesian Transfer Learning: An Overview of Probabilistic Graphical Models for Transfer Learning
- On the Statistical Consistency of Risk-Sensitive Bayesian Decision-Making
- Multifidelity Bayesian Optimization for Binomial Output
- On Disentanglement in Gaussian Process Variational Autoencoders
- mmFall: Fall Detection using 4D MmWave Radar and a Hybrid Variational RNN AutoEncoder
- Consistency of variational Bayes inference for estimation and model selection in mixtures
- Informative Bayesian model selection for RR Lyrae star classifiers
- Inf-VAE
- Bayesian multiscale deep generative model for the solution of high-dimensional inverse problems
- Unifying and generalizing models of neural dynamics during decision-making
- Inferential Wasserstein Generative Adversarial Networks
- A Probabilistic Representation of Deep Learning
- Neighborhood Cognition Consistent Multi-Agent Reinforcement Learning
- Towards Amortized Ranking-Critical Training for Collaborative Filtering
- Detecting and Tracking Communal Bird Roosts in Weather Radar Data
- Augmenting and Tuning Knowledge Graph Embeddings
- Learning Variational Word Masks to Improve the Interpretability of Neural Text Classifiers
- Bayesian neural networks at scale: a performance analysis and pruning study
- Learning to Solve AC Optimal Power Flow by Differentiating through Holomorphic Embeddings
- Topic Memory Networks for Short Text Classification
- Semi-supervised Stochastic Multi-Domain Learning using Variational Inference
- Non-linear regression models for behavioral and neural data analysis
- Non-linear process convolutions for multi-output Gaussian processes
- Posterior Dispersion Indices
- Local convexity of the TAP free energy and AMP convergence for Z2-synchronization
- On the Beta Prime Prior for Scale Parameters in High-Dimensional Bayesian Regression Models
- Viscos Flows: Variational Schur Conditional Sampling With Normalizing Flows
- Using Large Ensembles of Control Variates for Variational Inference
- A Variational Inference Algorithm for BKMR in the Cross-Sectional\n Setting
- Dirichlet Simplex Nest and Geometric Inference
- Learnable Bernoulli Dropout for Bayesian Deep Learning
Related