2019/02/05 by Prateek Jaiswal, Vinayak Rao, Vinayak A. Rao +4 · 2 citations
Computer Science · Mathematics · #Bayesian Methods and Mixture Models #FOS: Computer and information sciences #FOS: Mathematics #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Mathematical Approximation and Integration #Spectral Theory in Mathematical Physics #Statistical Methods and Inference #Statistics Theory (math.ST) #advanced mathematical theories #cs.LG #math.ST #stat.ML #stat.TH
paper · pdf · doi:10.48550/arxiv.1902.01902
openalex publication_date 2019/02/05 · openalex created_date 2019/04/11 · arxiv created 2020/08/14 · arxiv updated 2020/08/17 · openalex updated_date 2026/07/28
We study the asymptotic consistency properties of α-Rényi approximate posteriors, a class of variational Bayesian methods that approximate an intractable Bayesian posterior with a member of a tractable family of distributions, the member chosen to minimize the α-Rényi divergence from the true posterior. Unique to our work is that we consider settings with α> 1, resulting in approximations that upperbound the log-likelihood, and consequently have wider spread than traditional variational approaches that minimize the Kullback-Liebler (KL) divergence from the posterior. Our primary result identifies sufficient conditions under which consistency holds, centering around the existence of a 'good' sequence of distributions in the approximating family that possesses, among other properties, the right rate of convergence to a limit distribution. We further characterize the good sequence by demonstrating that a sequence of distributions that converges too quickly cannot be a good sequence. We also extend our analysis to the setting where α equals one, corresponding to the minimizer of the reverse KL divergence, and to models with local latent variables. We also illustrate the existence of good sequence with a number of examples. Our results complement a growing body of work focused on the frequentist properties of variational Bayesian methods.