2019/06/17 by Alexander Immer, Immer, Alexander, Guillaume Dehaene +2
Computer Science · Mathematics · #Computation (stat.CO) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Generative Adversarial Networks and Image Synthesis #Machine Learning (stat.ML) #Stochastic Gradient Optimization Techniques #stat.CO #stat.ML
paper · pdf · doi:10.48550/arxiv.1906.06914
Under review (NEURIPS 2019)
arxiv created 2019/06/17 · openalex publication_date 2019/06/17 · arxiv updated 2019/06/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
The Black Box Variational Inference (Ranganath et al. (2014)) algorithm provides a universal method for Variational Inference, but taking advantage of special properties of the approximation family or of the target can improve the convergence speed significantly. For example, if the approximation family is a transformation family, such as a Gaussian, then switching to the reparameterization gradient (Kingma and Welling (2014)) often yields a major reduction in gradient variance. Ultimately, reducing the variance can reduce the computational cost and yield better approximations. We present a new method to extend the reparameterization trick to more general exponential families including the Wishart, Gamma, and Student distributions. Variational Inference with Numerical Derivatives (VIND) approximates the gradient with numerical derivatives and reduces its variance using a tight coupling of the approximation family. The resulting algorithm is simple to implement and can profit from widely known couplings. Our experiments confirm that VIND effectively decreases the gradient variance and therefore improves the posterior approximation in relevant cases. It thus provides an efficient yet simple Variational Inference method for computing non-Gaussian approximations.