Core Lexicon and Contagious Words

2003/03/21 by Dmitri Volchenkov, Volchenkov, Dmitri, Philippe Blanchard +3
Physics and Astronomy · #Disordered Systems and Neural Networks (cond-mat.dis-nn) #FOS: Physical sciences #cond-mat.dis-nn

paper · pdf · doi:10.48550/arxiv.cond-mat/0303454

RevTex, 4 pages, 2 figures

arxiv created 2003/03/21 · arxiv updated 2009/11/30

Abstract

We present the new empirical parameter f_c, the most probable usage frequency of a word in a language, computed via the distribution of documents over frequency x of the word. This parameter allows for filtering the core lexicon of a language from the content words, which tend to be extremely frequent in some texts written in specific genres or by certain authors. Distributions of documents over frequencies for such words display long tails as x>f_c representing a bunch of documents in which such words are used in abundance. Collections of such documents exhibit a percolation like phase transition as the coarse grain of frequency Δf (flattening out the strongly irregular frequency data series) approaches the critical value f_c.

Related

• The empirical structure of word frequency distributions
• Neutral evolution and turnover over centuries of English word popularity
• Is language evolution grinding to a halt? The scaling of lexical turbulence in English fiction suggests it is not
• Lexical Co-occurrence, Statistical Significance, and Word Association
• Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Times