Stochastic Variational Inference

2012/06/29 by Matt Hoffman, Hoffman, Matt, David M. Blei +5 · 49 citations
Computer Science · Mathematics · #Artificial Intelligence (cs.AI) #Bayesian Methods and Mixture Models #Computation (stat.CO) #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (stat.ML) #Methodology (stat.ME) #Statistical Methods and Inference

paper · pdf · doi:10.48550/arxiv.1206.7051

openalex publication_date 2012/06/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We develop stochastic variational inference, a scalable algorithm for approximating posterior distributions. We develop this technique for a large class of probabilistic models and we demonstrate it with two probabilistic topic models, latent Dirichlet allocation and the hierarchical Dirichlet process topic model. Using stochastic variational inference, we analyze several large collections of documents: 300K articles from Nature, 1.8M articles from The New York Times, and 3.8M articles from Wikipedia. Stochastic inference can easily handle data sets of this size and outperforms traditional variational inference, which can only handle a smaller subset. (We also show that the Bayesian nonparametric topic model outperforms its parametric counterpart.) Stochastic variational inference lets us apply complex Bayesian models to massive data sets.

Citations

• On Information and Sufficiency
• Natural Gradient Works Efficiently in Learning
• A Stochastic Approximation Method
• The perceptron: A probabilistic model for information storage and organization in the brain.
• Bayesian Density Estimation and Inference Using Mixtures
• Variational Bayesian Inference with Stochastic Search
• A tutorial on hidden Markov models and selected applications in speech recognition
• "General Intelligence," Objectively Determined and Measured
• Probabilistic topic models

Cited by

• PYPM-GGD: Pitman-Yor Process Mixture with Generalized Gaussian Density using ADAM
• Bend to Mend: Toward Trustworthy Variational Bayes with Valid Uncertainty Quantification
• Distributed and recursive Bayesian inference for Big Data and complex spatio-temporal models
• Rethinking Likelihood distributions: Student's t Likelihood Boosts Bayesian Neural Network Performance
• Generative Actor Critic
• From Overfitting to Reliability: Introducing the Hierarchical Approximate Bayesian Neural Network
• Variational Inference for Fully Bayesian Hierarchical Linear Models
• Uncertainty-Preserving QBNNs: Multi-Level Quantization of SVI-Based Bayesian Neural Networks for Image Classification
• A Bayesian latent class reinforcement learning framework to capture adaptive, feedback-driven travel behaviour
• Meta-Learning Multi-armed Bandits for Beam Tracking in 5G and 6G Networks
• Credal Graph Neural Networks
• Uncertainty Reasoning with Photonic Bayesian Machines
• Accelerated Execution of Bayesian Neural Networks using a Single Probabilistic Forward Pass and Code Generation
• Optimising Density Computations in Probabilistic Programs via Automatic Loop Vectorisation
• Stein-based Optimization of Sampling Distributions in Model Predictive Path Integral Control
• Online and Interactive Bayesian Inference Debugging
• Bayesian Inference via GeTe _<i>x</i> ‐OTS Based Stochastic Synapse for Uncertainty‐Aware Medical Diagnostics
• A Black Box Variational Inference Scheme for Inverse Problems with Demanding Physics-Based Models
• Transformers can do Bayesian Clustering
• VALA: Learning Latent Anchors for Training-Free and Temporally Consistent
• Generative AI in Depth: A Survey of Recent Advances, Model Variants, and Real-World Applications
• Natural Gradient VI: Guarantees for Non-Conjugate Models
• A Frequentist Statistical Introduction to Variational Inference, Autoencoders, and Diffusion Models
• Geometric Convergence Analysis of Variational Inference via Bregman Divergences
• nuGPR: GPU-Accelerated Gaussian Process Regression with Iterative Algorithms and Low-Rank Approximations
• Reinforced sequential Monte Carlo for amortised sampling
• Automated Variational Inference in Probabilistic Programming
• Latent Uncertainty Representations for Video-based Driver Action and Intention Recognition
• Extracting the Structure of Press Releases for Predicting Earnings Announcement Returns
• A Hierarchical Variational Graph Fused Lasso for Recovering Relative Rates in Spatial Compositional Data
• Scalable Mean-Field Variational Inference via Preconditioned Primal-Dual Optimization
• Bayesian Neural Networks versus deep ensembles for uncertainty quantification in machine learning interatomic potentials
• Diffusion Bridge Variational Inference for Deep Gaussian Processes
• Information Geometry of Variational Bayes
• Reproducible workflow for online AI in digital health
• High-Dimensional Bayesian Model Comparison in Cosmology with GPU-accelerated Nested Sampling and Neural Emulators
• Is the `Agent' Paradigm a Limiting Framework for Next-Generation Intelligent Systems?
• Adaptive Bayesian computation for efficient biobank-scale genomic inference
• Kalman Bayesian Transformer
• Constrained Variational Inference via Safe Particle Flow
• Quantifying Out-of-Training Uncertainty of Neural-Network based Turbulence Closures
• Towards real-time surrogate-free Bayesian inversion for neutron reflectometry
• Spherical latent space models for social network analysis
• QUEENS: An Open-Source Python Framework for Solver-Independent Analyses of Large-Scale Computational Models
• Unbiased Stochastic Optimization for Gaussian Processes on Finite Dimensional RKHS
• Hi-fi functional priors by learning activations
• Bayesian Neural Network Surrogates for Bayesian Optimization of Carbon Capture and Storage Operations
• Integrated in vivo combinatorial functional genomics and spatial transcriptomics of tumours to decode genotype-to-phenotype relationships
• Large-scale entity resolution via microclustering Ewens--Pitman random partitions
• Stochastic Backpropagation and Approximate Inference in Deep Generative\n Models

Related

• Stochastic Collapsed Variational Bayesian Inference for Latent Dirichlet Allocation
• Sparse Stochastic Inference for Latent Dirichlet allocation
• Stochastic Collapsed Variational Inference for Sequential Data
• Variational Inference: A Review for Statisticians
• Variational Inference: A Review for Statisticians