Types, Tokens, and Hapaxes: A New Heap's Law

2018/12/31 by Davis, Victor · 2 citations
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.1901.00521

Abstract

Heap's Law states that in a large enough text corpus, the number of types as a function of tokens grows as N=KM^β for some free parameters K,β. Much has been written about how this result and various generalizations can be derived from Zipf's Law. Here we derive from first principles a completely novel expression of the type-token curve and prove its superior accuracy on real text. This expression naturally generalizes to equally accurate estimates for counting hapaxes and higher n-legomena.

Cited by

• From Zipf's Law to Neural Scaling through Heaps' Law and Hilberg's Hypothesis
• Quadratic Term Correction on Heaps' Law

Related

• Proper Interpretation of Heaps' and Zipf's Laws
• Corrections of Zipf's and Heaps' Laws Derived from Hapax Rate Models
• Quadratic Term Correction on Heaps' Law
• A statistical test for the Zipf's law by deviations from the Heaps' law
• Complete asymptotic type-token relationship for growing complex systems with inverse power-law count rankings