I-Divergence Geometry of Probability Distributions and Minimization Problems

1975/02/01 by Imre Csiszár · 18 citations
Physics and Astronomy · Mathematics · #Statistical Mechanics and Entropy #Advanced Statistical Methods and Models #Mathematical Inequalities and Applications #Mathematics #Lagrange multiplier #Euclidean geometry #Divergence (linguistics) #Regular polygon #Generalization #Minification #Kullback–Leibler divergence #Convergence (economics) #Geometric probability #Multiplier (economics) #Combinatorics #Applied mathematics #Mathematical analysis #Mathematical optimization #Geometry #Statistics

paper · pdf · doi:10.1214/aop/1176996454

openalex publication_date 1975/02/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/29

Abstract

Some geometric properties of PD's are established, Kullback's I-divergence playing the role of squared Euclidean distance. The minimum discrimination information problem is viewed as that of projecting a PD onto a convex set of PD's and useful existence theorems for and characterizations of the minimizing PD are arrived at. A natural generalization of known iterative algorithms converging to the minimizing PD in special situations is given; even for those special cases, our convergence proof is more generally valid than those previously published. As corollaries of independent interest, generalizations of known results on the existence of PD's or nonnegative matrices of a certain form are obtained. The Lagrange multiplier technique is not used.

Citations

• An Iterative Procedure for Estimation in Contingency Tables
• Information Theory and Statistical Mechanics

Cited by

• Large deviations of the trajectory of empirical distributions of Feller processes on locally compact spaces
• Convex programming in optimal control and information theory
• Amplification at Equilibrium: Structural and Thermodynamic Limitations, and Implementation
• Conditioned Generative Modeling of Molecular Glues: A Realistic AI Approach for Synthesizable Drug-like Molecules
• Rényi Divergence and Kullback-Leibler Divergence
• Maximum likelihood estimators uniformly minimize distribution variance among distribution unbiased estimators in exponential families
• Still No Free Lunches: The Price to Pay for Tighter PAC-Bayes Bounds
• Minimum description length principle for maximum entropy model selection
• Entropic regularization of Wasserstein distance between infinite-dimensional Gaussian measures and Gaussian processes
• Unifying quantum measurement constructions via a relative-entropy minimum change principle
• Regularity properties and pathologies of position-space renormalization-group transformations: Scope and limitations of Gibbsian theory
• Composition of Probability Measures on Finite Spaces
• Nonextensive triangle equality and other properties of Tsallis relative-entropy minimization
• A Localization Approach to Improve Iterative Proportional Scaling in Gaussian Graphical Models
• Equivalence of ensembles for two-species zero-range invariant measures
• On the Approximation of a Discrete Multivariate Probability Distribution Using the New Concept of t-Cherry Junction Tree
• Information Theory and Statistics: A Tutorial
• Kullback–Leibler divergence [wikipedia]