EigenVI: score-based variational inference with orthogonal function expansions

2024/10/31 by Diana Cai, Cai, Diana, Chirag Modi +10 · 2 voices · 7 citations
Computer Science · Mathematics · Physics and Astronomy · #Applied mathematics #Artificial intelligence #Biology #Computer science #Domain Adaptation and Few-Shot Learning #Function (biology) #Generative Adversarial Networks and Image Synthesis #Inference #Mathematics #Model Reduction and Neural Networks #Score #Statistics

paper · pdf · doi:10.48550/arxiv.2410.24054

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/10/31 · openalex created_date 2024/11/14 · openalex updated_date 2026/08/01

Abstract

We develop EigenVI, an eigenvalue-based approach for black-box variational inference (BBVI). EigenVI constructs its variational approximations from orthogonal function expansions. For distributions over ℝ^D, the lowest order term in these expansions provides a Gaussian variational approximation, while higher-order terms provide a systematic way to model non-Gaussianity. These approximations are flexible enough to model complex distributions (multimodal, asymmetric), but they are simple enough that one can calculate their low-order moments and draw samples from them. EigenVI can also model other types of random variables (e.g., nonnegative, bounded) by constructing variational approximations from different families of orthogonal functions. Within these families, EigenVI computes the variational approximation that best matches the score function of the target distribution by minimizing a stochastic estimate of the Fisher divergence. Notably, this optimization reduces to solving a minimum eigenvalue problem, so that EigenVI effectively sidesteps the iterative gradient-based optimizations that are required for many other BBVI algorithms. (Gradient-based methods can be sensitive to learning rates, termination criteria, and other tunable hyperparameters.) We use EigenVI to approximate a variety of target distributions, including a benchmark suite of Bayesian models from posteriordb. On these distributions, we find that EigenVI is more accurate than existing methods for Gaussian BBVI.

Cited by

• How to Square Tensor Networks and Circuits Without Squaring Them
• Generalized Guarantees for Variational Inference in the Presence of Even and Elliptical Symmetry
• Large-scale empirical tuning and comparison of default optimizers for variational inference
• Fisher meets Feynman: score-based variational inference with a product of experts
• To MCMC or not to MCMC: Evaluating non-MCMC methods for Bayesian penalized regression
• VHU-Net: Variational Hadamard U-Net for Body MRI Bias Field Correction
• Variational Inference with Mixtures of Isotropic Gaussians

Discussions

• Heading to #NeurIPS2024 next week to present EigenVI: score-based variational inference with orthogonal function expansions Link: arxiv.org/abs/2410.24054 Poster: Friday 7:30pm, East Exhibit Hall A-C [bsky, 41 points, 1 comments]
• Co-authors: @dianarycai.bsky.social, Chirag Modi, Robert Gower, David Blei, and Lawrence Saul. Link: arxiv.org/abs/2410.24054 Poster: Friday 7:30pm, East Exhibit Hall A-C #3900 [bsky, 0 points, 0 comments]

Related

• Batch, match, and patch: low-rank approximations for score-based variational inference
• Variational Inference with Gaussian Score Matching
• Semi-Implicit Variational Inference
• On the Convergence of Black-Box Variational Inference
• Batch and match: black-box variational inference with a score-based divergence