2022/10/06 by Watts, Stephen, Lisa Crow, Crow, Lisa
Computer Science · #Applications (stat.AP) #Data Analysis #FOS: Computer and information sciences #FOS: Physical sciences #Methodology (stat.ME) #Neural Networks and Applications #Statistics and Probability (physics.data-an)
paper · pdf · doi:10.48550/arxiv.2210.02848
openalex publication_date 2022/10/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
The histogram is a key method for visualizing data and estimating the underlying probability distribution. Incorrect conclusions about the data result from over or under-binning. A new method based on the Shannon entropy of the histogram uses a simple formula based on the differential entropy estimated from nearest-neighbour distances. Links are made between the new method and other algorithms such as Scott's formula, and cost and risk function methods. A parameter is found that predicts over and under-binning, which can be estimated for any histogram. The new algorithm is shown to be robust by application to real data.