Pragmatic inference of scalar implicature by LLMs

2024/08/13 by Ye-eun Cho, Cho, Ye-eun, Seong mook Kim +1 · 3 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · #Biomedical Text Mining and Ontologies #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Semantic Web and Ontologies

paper · pdf · doi:10.48550/arxiv.2408.06673

openalex publication_date 2024/08/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This study investigates how Large Language Models (LLMs), particularly BERT (Devlin et al., 2019) and GPT-2 (Radford et al., 2019), engage in pragmatic inference of scalar implicature, such as some. Two sets of experiments were conducted using cosine similarity and next sentence/token prediction as experimental methods. The results in experiment 1 showed that, both models interpret some as pragmatic implicature not all in the absence of context, aligning with human language processing. In experiment 2, in which Question Under Discussion (QUD) was presented as a contextual cue, BERT showed consistent performance regardless of types of QUDs, while GPT-2 encountered processing difficulties since a certain type of QUD required pragmatic inference for implicature. The findings revealed that, in terms of theoretical approaches, BERT inherently incorporates pragmatic implicature not all within the term some, adhering to Default model (Levinson, 2000). In contrast, GPT-2 seems to encounter processing difficulties in inferring pragmatic implicature within context, consistent with Context-driven model (Sperber and Wilson, 2002).

Cited by

• Evaluating Communicative Belief Updates in Large Language Models via Implicature Recognition and Cancellation
• Implicature in Interaction: Understanding Implicature Improves Alignment in Human-LLM Interaction
• Are BabyLMs Deaf to Gricean Maxims? A Pragmatic Evaluation of Sample-efficient Language Models

Related

• The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs
• Experimental Pragmatics with Machines: Testing LLM Predictions for the Inferences of Plain and Embedded Disjunctions
• Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition
• The role of scalar diversity and question under discussion in deriving impli- catures with embedded scales
• Probing Large Language Models for Scalar Adjective Lexical Semantics and Scalar Diversity Pragmatics