2010/09/01 by Frank Nielsen, Richard Nock · 5 citations
Computer Science · Physics and Astronomy · Mathematics · #Image and Signal Denoising Methods #Advanced Image Processing Techniques #Statistical Mechanics and Entropy #Exponential family #Mathematics #Entropy (arrow of time) #Exponential function #Natural exponential family #Multinomial distribution #Poisson distribution #Dirichlet distribution #Statistical physics #Gaussian #Gamma distribution #Applied mathematics #Combinatorics #Statistics #Mathematical analysis #Physics
paper · doi:10.1109/icip.2010.5652054
openalex publication_date 2010/09/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/29
Statistical modeling of images plays a crucial role in modern image processing tasks like segmentation, object detection and restoration. Although Gaussian distributions are conveniently handled mathematically, the role of many other types of distributions has been revealed and emphasized by natural image statistics. In this paper, we consider a versatile class of distributions called exponential families that encompasses many well-known distributions, such as Gaussian, Poisson, multinomial, Gamma/Beta and Dirichlet distributions, just to name a few. For those families, we derive mathematical expressions for their Shannon entropy and cross-entropy, give a geometric interpretation, and show that they admit closed-form formula up to some entropic normalizing constant depending on the carrier measure but independent of the member of the family. This allows one to design algorithms that can compare exactly entropies and cross-entropies of exponential family distributions although some of them have strictus sensus no known closed forms (eg., Poisson). We discuss about maximum entropy and touch upon the entropy of mixtures of exponential families for which we provide a relative entropy upper bound.