2024/09/16 by Mario Giulianelli, Giulianelli, Mario, Andreas Opedal +3 · 3 citations
Physics and Astronomy · Social Sciences · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Information Theory (cs.IT) #Opinion Dynamics and Social Influence #Social Media and Politics
paper · pdf · doi:10.48550/arxiv.2409.10728
openalex publication_date 2024/09/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
We introduce a generalization of classic information-theoretic measures of predictive uncertainty in online language processing, based on the simulation of expected continuations of incremental linguistic contexts. Our framework provides a formal definition of anticipatory and responsive measures, and it equips experimenters with the tools to define new, more expressive measures beyond standard next-symbol entropy and surprisal. While extracting these standard quantities from language models is convenient, we demonstrate that using Monte Carlo simulation to estimate alternative responsive and anticipatory measures pays off empirically: New special cases of our generalized formula exhibit enhanced predictive power compared to surprisal for human cloze completion probability as well as ELAN, LAN, and N400 amplitudes, and greater complementarity with surprisal in predicting reading times.