CLAM: Selective Clarification for Ambiguous Questions with Generative Language Models

2022/12/15 by Lorenz Kuhn, Yarin Gal, Kuhn, Lorenz +3 · 26 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Speech and dialogue systems #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2212.07769

openalex publication_date 2022/12/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Users often ask dialogue systems ambiguous questions that require clarification. We show that current language models rarely ask users to clarify ambiguous questions and instead provide incorrect answers. To address this, we introduce CLAM: a framework for getting language models to selectively ask for clarification about ambiguous user questions. In particular, we show that we can prompt language models to detect whether a given question is ambiguous, generate an appropriate clarifying question to ask the user, and give a final answer after receiving clarification. We also show that we can simulate users by providing language models with privileged information. This lets us automatically evaluate multi-turn clarification dialogues. Finally, CLAM significantly improves language models' accuracy on mixed ambiguous and unambiguous questions relative to SotA.

Cited by

• NRR-Core: Non-Resolution Reasoning as a Computational Framework for Contextual Identity and Ambiguity Preservation
• MAC: A Multi-Agent Framework for Interactive User Clarification in Multi-turn Conversations
• Learning Steerable Clarification Policies with Collaborative Self-play
• Reasoning With a Star: A Heliophysics Dataset and Benchmark for Agentic Scientific Reasoning
• SG-CoT: An Ambiguity-Aware Robotic Planning Framework using Scene Graph Representations
• Identifying & Interactively Refining Ambiguous User Goals for Data Visualization Code Generation
• CaRT: Teaching LLM Agents to Know When They Know Enough
• It Depends: Resolving Referential Ambiguity in Minimal Contexts with Commonsense Knowledge
• Sparse Neurons Carry Strong Signals of Question Ambiguity in LLMs
• Can Multiple Responses from an LLM Reveal the Sources of Its Uncertainty?
• Beyond Solving Math Quiz: Evaluating the Ability of Large Reasoning Models to Ask for Information
• MAQuA: Adaptive Question-Asking for Multidimensional Mental Health Screening using Item Response Theory
• Demystifying Feature Requests: Leveraging LLMs to Refine Feature Requests in Open-Source Software
• First Ask Then Answer: A Framework Design for AI Dialogue Based on Supplementary Questioning with Large Language Models
• Beyond Passive Critical Thinking: Fostering Proactive Questioning to Enhance Human-AI Collaboration
• Referential ambiguity and clarification requests: comparing human and LLM behaviour
• From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centered
• A Preference-Driven Methodology for High-Quality Solidity Code Generation
• ChemAU: Harness the Reasoning of LLMs in Chemical Research with Adaptive Uncertainty Estimation
• Agents Require Metacognitive and Strategic Reasoning to Succeed in the Coming Labor Markets
• What Prompts Don't Say: Understanding and Managing Underspecification in LLM Prompts
• Disambiguation in Conversational Question Answering in the Era of LLMs and Agents: A Survey
• Spilled Energy in Large Language Models
• Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?
• Value of Information: A Framework for Human-Agent Communication
• From Misleading Queries to Accurate Answers: A Three-Stage Fine-Tuning Method for LLMs

Related

• ClarifyMT-Bench: Benchmarking and Improving Multi-Turn Clarification for Conversational Large Language Models
• Tree of Clarifications: Answering Ambiguous Questions with Retrieval-Augmented Large Language Models
• Open-domain clarification question generation without question examples
• CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models
• Interactive Question Clarification in Dialogue via Reinforcement Learning