2021/09/24 by Isaac J. Sledge, José C. Prı́ncipe, Sledge, Isaac J. +1
Physics and Astronomy · Decision Sciences · Engineering · #Statistical Mechanics and Entropy #Probabilistic and Robust Engineering Design #Control Systems and Identification
paper · pdf · doi:10.48550/arxiv.2109.11737
Conventional information-theoretic quantities assume access to probability distributions. Estimating such distributions is not trivial. Here, we consider function-based formulations of cross entropy that sidesteps this a priori estimation requirement. We propose three measures of Rényi's α-cross-entropies in the setting of reproducing-kernel Hilbert spaces. Each measure has its appeals. We prove that we can estimate these measures in an unbiased, non-parametric, and minimax-optimal way. We do this via sample-constructed Gram matrices. This yields matrix-based estimators of Rényi's α-cross-entropies. These estimators satisfy all of the axioms that Rényi established for divergences. Our cross-entropies can thus be used for assessing distributional differences. They are also appropriate for handling high-dimensional distributions, since the convergence rate of our estimator is independent of the sample dimensionality. Python code for implementing these measures can be found at https://github.com/isledge/MBRCE