Tighter Variational Representations of f-Divergences via Restriction to Probability Measures

2012/06/18 by Avraham Ruderman, Mark D. Reid, Ruderman, Avraham +4
Mathematics · Physics and Astronomy · #Advanced Statistical Methods and Models #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Statistical Mechanics and Entropy #Statistical Methods and Inference

paper · pdf · doi:10.48550/arxiv.1206.4664

openalex publication_date 2012/06/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We show that the variational representations for f-divergences currently used in the literature can be tightened. This has implications to a number of methods recently proposed based on this representation. As an example application we use our tighter representation to derive a general f-divergence estimator based on two i.i.d. samples and derive the dual program for this estimator that performs well empirically. We also point out a connection between our estimator and MMD.

Related

• f-Divergence Variational Inference
• A Lower Bound on Arbitrary f--Divergences in Terms of the Total Variation
• Optimizing Variational Representations of Divergences and Accelerating their Statistical Estimation
• The f-Divergence Expectation Iteration Scheme
• Alpha-Beta Divergence For Variational Inference