2018/05/22 by Andrew D. Back, Back, Andrew D., Daniel Angus +3
Computer Science · #Data Analysis #FOS: Computer and information sciences #FOS: Mathematics #FOS: Physical sciences #Information Theory (cs.IT) #Neural Networks and Applications #Statistics Theory (math.ST) #Statistics and Probability (physics.data-an)
paper · pdf · doi:10.48550/arxiv.1805.08929
openalex publication_date 2018/05/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Calculating the Shannon entropy for symbolic sequences has been widely\nconsidered in many fields. For descriptive statistical problems such as\nestimating the N-gram entropy of English language text, a common approach is to\nuse as much data as possible to obtain progressively more accurate estimates.\nHowever in some instances, only short sequences may be available. This gives\nrise to the question of how many samples are needed to compute entropy. In this\npaper, we examine this problem and propose a method for estimating the number\nof samples required to compute Shannon entropy for a set of ranked symbolic\nnatural events. The result is developed using a modified Zipf-Mandelbrot law\nand the Dvoretzky-Kiefer-Wolfowitz inequality, and we propose an algorithm\nwhich yields an estimate for the minimum number of samples required to obtain\nan estimate of entropy with a given confidence level and degree of accuracy.\n