2023/05/17 by Kenneth Bogert, Bogert, Kenneth, Matthew Kothe +1
Computer Science · Neuroscience · Physics and Astronomy · #Computer Vision and Pattern Recognition (cs.CV) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Information Theory (cs.IT) #Machine Learning (cs.LG) #Neural dynamics and brain function #Statistical Mechanics and Entropy
paper · pdf · doi:10.48550/arxiv.2305.09868
openalex publication_date 2023/05/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
The Principle of Maximum Entropy is a rigorous technique for estimating an unknown distribution given partial information while simultaneously minimizing bias. However, an important requirement for applying the principle is that the available information be provided error-free (Jaynes, 1982). We relax this requirement using a memoryless communication channel as a framework to derive a new, more general principle. We show our new principle provides an upper bound on the entropy of the unknown distribution and the amount of information lost due to the use of a given communications channel is unknown unless the unknown distribution's entropy is also known. Using our new principle we provide a new interpretation of the classic principle and experimentally show its performance relative to the classic principle and some other generally applicable solutions.