Understand the Implication: Learning to Think for Pragmatic Understanding

2025/06/16 by Settaluri Lakshmi Sravanthi, Sravanthi, Settaluri Lakshmi, Kishan Maharaj +7
Arts and Humanities · Social Sciences · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #EFL/ESL Teaching and Learning #Education and Critical Thinking Development #FOS: Computer and information sciences #Second Language Learning and Teaching

paper · pdf · doi:10.48550/arxiv.2506.13559

openalex publication_date 2025/06/16 · openalex created_date 2025/10/13 · openalex updated_date 2026/07/28

Abstract

Pragmatics, the ability to infer meaning beyond literal interpretation, is crucial for social cognition and communication. While LLMs have been benchmarked for their pragmatic understanding, improving their performance remains underexplored. Existing methods rely on annotated labels but overlook the reasoning process humans naturally use to interpret implicit meaning. To bridge this gap, we introduce a novel pragmatic dataset, ImpliedMeaningPreference, that includes explicit reasoning (thoughts) for both correct and incorrect interpretations. Through preference-tuning and supervised fine-tuning, we demonstrate that thought-based learning significantly enhances LLMs' pragmatic understanding, improving accuracy by 11.12% across model families. We further discuss a transfer-learning study where we evaluate the performance of thought-based training for the other tasks of pragmatics (presupposition, deixis) that are not seen during the training time and observe an improvement of 16.10% compared to label-trained models.

Citations

• The Llama 3 Herd of Models
• Gemma 2: Improving Open Language Models at a Practical Size
• Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective
• Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive
• PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilities
• System 2 Attention (is something you might need too)
• Large Language Models: The Need for Nuance in Current Debates and a Pragmatic Perspective on Understanding
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• GPT-4 Technical Report
• A fine-grained comparison of pragmatic language understanding in humans and language models
• The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs
• Improving alignment of dialogue agents via targeted human judgements
• Training Verifiers to Solve Math Word Problems
• Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies
• "I'd rather just go to bed": Understanding Indirect Answers
• Language Models are Few-Shot Learners
• Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature\n and PRESupposition
• Harnessing the linguistic signal to predict scalar inferences
• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• An overview of gradient descent optimization algorithms
• SQUINKY! A Corpus of Sentence-level Formality, Informativeness, and Implicature
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• Pragmatic Theories Enhance Understanding of Implied Meanings in LLMs
• The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs
• The Pragmatic Mind of Machines: Tracing the Emergence of Pragmatic Competence in Large Language Models
• On the Same Wavelength? Evaluating Pragmatic Reasoning in Language Models across Broad Concepts
• Thinking Before You Speak: A Proactive Test-time Scaling Approach