f-divergence Inequalities

2015/08/31 by Igal Sason, Sergio Verdú · 7 citations
Computer Science · Mathematics · #cs.IT #math.IT #math.PR

paper · pdf · doi:10.1109/tit.2016.2603151

IEEE Trans. on Information Theory, vol. 62, no. 11, pp. 5973--6006, November 2016. This manuscript is identical to the journal paper, apart of some additional material which includes Sections III-C and IV-F, and three technical proofs

arxiv created 2016/12/04 · arxiv updated 2016/12/06

Abstract

This paper develops systematic approaches to obtain f-divergence inequalities, dealing with pairs of probability measures defined on arbitrary alphabets. Functional domination is one such approach, where special emphasis is placed on finding the best possible constant upper bounding a ratio of f-divergences. Another approach used for the derivation of bounds among f-divergences relies on moment inequalities and the logarithmic-convexity property, which results in tight bounds on the relative entropy and Bhattacharyya distance in terms of χ² divergences. A rich variety of bounds are shown to hold under boundedness assumptions on the relative information. Special attention is devoted to the total variation distance and its relation to the relative information and relative entropy, including "reverse Pinsker inequalities," as well as on the E_γ divergence, which generalizes the total variation distance. Pinsker's inequality is extended for this type of f-divergence, a result which leads to an inequality linking the relative entropy and relative information spectrum. Integral expressions of the Rényi divergence in terms of the relative information spectrum are derived, leading to bounds on the Rényi divergence in terms of either the variational distance or relative entropy.

Cited by

• Diff2SP: Diffusion Models for Correlated Scenario Generation in Stochastic Programming
• Multi-Marginal Couplings for Metropolis-Hastings
• Closing the Indexing-Decoding Gap in Multimodal Generative Retrieval via Prefix Retention Optimization
• Seeking Flat Minima over Diverse Surrogates for Improved Adversarial Transferability: A Theoretical Framework and Algorithmic Instantiation
• Transport f divergences
• Thermodynamic Uncertainty Relation for f-divergence Entropy Production
• Fast Rates for Inverse Reinforcement Learning

Related

• f-Divergence Inequalities via Functional Domination
• Upper Bounds on the Relative Entropy and Rényi Divergence as a Function of Total Variation Distance for Finite Alphabets
• Sharp Inequalities for f-divergences
• On Reverse Pinsker Inequalities
• Bounds on Non-Symmetric Divergence Measures in Terms of Symmetric Divergence Measures