Can LLMs Ground when they (Don't) Know: A Study on Direct and Loaded Political Questions

2025/06/10 by Clara Lachenmaier, Lachenmaier, Clara, Judith Sieker +3 · 1 citation
Computer Science · Social Sciences · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Expert finding and Q&A systems #FOS: Computer and information sciences #Misinformation and Its Impacts #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2506.08952

openalex publication_date 2025/06/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Communication among humans relies on conversational grounding, allowing interlocutors to reach mutual understanding even when they do not have perfect knowledge and must resolve discrepancies in each other's beliefs. This paper investigates how large language models (LLMs) manage common ground in cases where they (don't) possess knowledge, focusing on facts in the political domain where the risk of misinformation and grounding failure is high. We examine the ability of LLMs to answer direct knowledge questions and loaded questions that presuppose misinformation. We evaluate whether loaded questions lead LLMs to engage in active grounding and correct false user beliefs, in connection to their level of knowledge and their political bias. Our findings highlight significant challenges in LLMs' ability to engage in grounding and reject false user beliefs, raising concerns about their role in mitigating misinformation in political discourse.

Citations

• Defining Knowledge: Bridging Epistemology and Large Language Models
• The Llama 3 Herd of Models
• The Illusion of Competence: Evaluating the Effect of Explanations on Users' Mental Models of Visual Question Answering Systems
• Measuring Political Bias in Large Language Models: What Is Said and How It Is Said
• Syn-QA2: Evaluating False Assumptions in Long-tail Questions with Synthetic QA Datasets
• Knowledge Conflicts for LLMs: A Survey
• Pregnant Questions: The Importance of Pragmatic Awareness in Maternal Health Question Answering
• Grounding Gaps in Language Model Generations
• No that's not what I meant: Handling Third Position Repair in Conversational Question Answering
• From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models
• The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation
• (QA)²: Question Answering with Questionable Assumptions
• Discovering Language Model Behaviors with Model-Written Evaluations
• CREPE: Open-Domain Question Answering with False Presuppositions
• The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs
• Grounding 'Grounding' in NLP
• Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering
• Reducing conversational agents' overconfidence through linguistic calibration
• Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition

Cited by

• Are BabyLMs Deaf to Gricean Maxims? A Pragmatic Evaluation of Sample-efficient Language Models

Related

• The Earth is Flat because...: Investigating LLMs' Belief towards Misinformation via Persuasive Conversation
• LLMs Struggle to Reject False Presuppositions when Misinformation Stakes are High
• The Political Preferences of LLMs
• How LLMs Fail to Support Fact-Checking
• Debating with More Persuasive LLMs Leads to More Truthful Answers