1995/05/01 by Eric Sven Ristad, Ristad, Eric Sven, Robert G. Thomas +1
Biochemistry, Genetics and Molecular Biology · Computer Science · #Algorithms and Data Compression #Computation and Language (cs.CL) #DNA and Biological Computing #FOS: Computer and information sciences #Machine Learning and Algorithms #cmp-lg #cs.CL
paper · pdf · doi:10.48550/arxiv.cmp-lg/9505002
8 pages, to appear in Proc. ACL 1995
arxiv created 1995/05/01 · openalex publication_date 1995/05/01 · arxiv updated 2009/11/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
We introduce three new techniques for statistical language models: extension modeling, nonmonotonic contexts, and the divergence heuristic. Together these techniques result in language models that have few states, even fewer parameters, and low message entropies. For example, our techniques achieve a message entropy of 1.97 bits/char on the Brown corpus using only 89,325 parameters. In contrast, the character 4-gram model requires more than 250 times as many parameters in order to achieve a message entropy of only 2.47 bits/char. The fact that our model performs significantly better while using vastly fewer parameters indicates that it is a better probability model of natural language text.