A fine-grained comparison of pragmatic language understanding in humans and language models

2022/12/13 by Jennifer J. Hu, Hu, Jennifer, Sammy Floyd +7 · 9 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech and dialogue systems #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2212.06801

openalex publication_date 2022/12/13 · openalex created_date 2022/12/27 · openalex updated_date 2026/07/28

Abstract

Pragmatics and non-literal language understanding are essential to human communication, and present a long-standing challenge for artificial language models. We perform a fine-grained comparison of language models and humans on seven pragmatic phenomena, using zero-shot prompting on an expert-curated set of English materials. We ask whether models (1) select pragmatic interpretations of speaker utterances, (2) make similar error patterns as humans, and (3) use similar linguistic cues as humans to solve the tasks. We find that the largest models achieve high accuracy and match human error patterns: within incorrect responses, models favor literal interpretations over heuristic-based distractors. We also find preliminary evidence that models and humans are sensitive to similar linguistic cues. Our results suggest that pragmatic behaviors can emerge in models without explicitly constructed representations of mental states. However, models tend to struggle with phenomena relying on social expectation violations.

Cited by

• You Prefer This One, I Prefer Yours: Using Reference Words is Harder Than Vocabulary Words for Humans and Multimodal Language Models
• Large language models are not about natural language
• QSTN: A Modular Framework for Robust Questionnaire Inference with Large Language Models
• Tracing the ongoing emergence of human-like reasoning in Large Language Models
• Cognitive Foundations for Reasoning and Their Manifestation in LLMs
• Quantification and object perception in Multimodal Large Language Models and human linguistic cognition
• Evaluating Language Models' Evaluations of Games
• Are BabyLMs Deaf to Gricean Maxims? A Pragmatic Evaluation of Sample-efficient Language Models
• Are Language Models Models?

Related

• On the Same Wavelength? Evaluating Pragmatic Reasoning in Language Models across Broad Concepts
• Are Language Models Worse than Humans at Following Prompts? It's Complicated
• The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs
• The Pragmatic Mind of Machines: Tracing the Emergence of Pragmatic Competence in Large Language Models
• Tracing the ongoing emergence of human-like reasoning in Large Language Models