Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?

2022/12/23 by Byung-Doh Oh, Oh, Byung-Doh, William Schuler +1 · 10 citations
Computer Science · Medicine · #Artificial Intelligence in Healthcare and Education #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2212.12131

openalex publication_date 2022/12/23 · openalex created_date 2023/01/04 · openalex updated_date 2026/07/28

Abstract

This work presents a detailed linguistic analysis into why larger Transformer-based pre-trained language models with more parameters and lower perplexity nonetheless yield surprisal estimates that are less predictive of human reading times. First, regression analyses show a strictly monotonic, positive log-linear relationship between perplexity and fit to reading times for the more recently released five GPT-Neo variants and eight OPT variants on two separate datasets, replicating earlier results limited to just GPT-2 (Oh et al., 2022). Subsequently, analysis of residual errors reveals a systematic deviation of the larger variants, such as underpredicting reading times of named entities and making compensatory overpredictions for reading times of function words such as modals and conjunctions. These results suggest that the propensity of larger Transformer-based models to 'memorize' sequences during training makes their surprisal estimates diverge from humanlike expectations, which warrants caution in using pre-trained language models to study human language processing.

Cited by

• Large language models are not about natural language
• Developmental Atlas of Attention Head Specialization: Spacing, Stranding, and the Capacity Tax of BPE Tokenization
• CLASS-IT: Conversational and Lecture-Aligned Small-Scale Instruction Tuning for BabyLMs
• Topoformer: brain-like topographic organization in Transformer language models through spatial querying and reweighting
• Are Language Models Models?
• LLMs Learn Constructions That Humans Do Not Know
• Correlates of Image Memorability in Vision Encoders: Activations, Attention Entropy, Patch Uniformity and Autoencoder Losses
• How do Humans and LLMs Process Confusing Code?
• A systematic evaluation of Dutch large language models’ surprisal estimates in sentence, paragraph and book reading
• Human-like fleeting memory improves language learning but impairs reading time prediction in transformer language models

Related

• Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens
• Surprisal from Larger Transformer-based Language Models Predicts fMRI Data More Poorly
• Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Times
• Temperature-scaling surprisal estimates improve fit to human reading times -- but does it do so for the "right reasons"?
• Linear Recency Bias During Training Improves Transformers' Fit to Reading Times