Prediction and Entropy of Printed English

1951/01/01 by C. E. Shannon, Claude E. Shannon · 8 citations
Computer Science · #Natural Language Processing Techniques #Neural Networks and Applications

paper · doi:10.1002/j.1538-7305.1951.tb01366.x

crossref issued 1951/01/01 · crossref published 1951/01/01 · crossref published-print 1951/01/01 · openalex publication_date 1951/01/01 · crossref published-online 2013/07/29 · crossref created 2013/07/29 · crossref deposited 2020/10/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/25 · crossref indexed 2026/08/01

Abstract

A new method of estimating the entropy and redundancy of a language is described. This method exploits the knowledge of the language statistics possessed by those who speak the language, and depends on experimental results in prediction of the next letter when the preceding text is known. Results of experiments in prediction are given, and some properties of an ideal predictor are developed.

Cited by

• The World’s Technological Capacity to Store, Communicate, and Compute Information
• The estimation of stochastic context-free grammars using the Inside-Outside algorithm
• The Brain Abstracted
• Dynamic Natural Language Processing with Recurrence Quantification Analysis
• Estimators for Multivariate Information Measures in General Probability Spaces
• Claude Elwood Shannon 30 April 1916 — 24 February 2001
• Measuring the Efficiency of the Intraday Forex Market with a Universal Data Compression Algorithm
• Probabilistic models of language processing and acquisition