f-Divergence Variational Inference

2020/09/28 by Wan, Neng, Li, Dapeng, Hovakimyan, Naira · 2 citations
#FOS: Computer and information sciences #Information Theory (cs.IT) #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.2009.13093

Abstract

This paper introduces the f-divergence variational inference (f-VI) that generalizes variational inference to all f-divergences. Initiated from minimizing a crafty surrogate f-divergence that shares the statistical consistency with the f-divergence, the f-VI framework not only unifies a number of existing VI methods, e.g. Kullback-Leibler VI, Rényi's α-VI, and χ-VI, but offers a standardized toolkit for VI subject to arbitrary divergences from f-divergence family. A general f-variational bound is derived and provides a sandwich estimate of marginal likelihood (or evidence). The development of the f-VI unfolds with a stochastic optimization scheme that utilizes the reparameterization trick, importance weighting and Monte Carlo approximation; a mean-field approximation scheme that generalizes the well-known coordinate ascent variational inference (CAVI) is also proposed for f-VI. Empirical examples, including variational autoencoders and Bayesian neural networks, are provided to demonstrate the effectiveness and the wide applicability of f-VI.

Cited by

• Accelerating Hamiltonian Monte Carlo for Bayesian Inference in Neural Networks and Neural Operators
• The Use of Variational Inference for Lifetime Data with Spatial Correlations

Related

• Tighter Variational Representations of f-Divergences via Restriction to Probability Measures
• Alpha-Beta Divergence For Variational Inference
• Rényi Divergence Variational Inference
• Perturbative Black Box Variational Inference
• Alpha-Divergences in Variational Dropout