2025/01/11 by Adam Nohejl, Taro Watanabe, Nohejl, Adam +1
Computer Science · Neuroscience · Psychology · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Neurobiology of Language and Bilingualism #Second Language Acquisition and Learning #Text Readability and Simplification
paper · pdf · doi:10.48550/arxiv.2501.06536
openalex publication_date 2025/01/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Various measures of dispersion have been proposed to paint a fuller picture of a word's distribution in a corpus, but only little has been done to validate them externally. We evaluate a wide range of dispersion measures as predictors of lexical decision time, word familiarity, and lexical complexity in five diverse languages. We find that the logarithm of range is not only a better predictor than log-frequency across all tasks and languages, but that it is also the most powerful additional variable to log-frequency, consistently outperforming the more complex dispersion measures. We discuss the effects of corpus part granularity and logarithmic transformation, shedding light on contradictory results of previous studies.