Tree of Clarifications: Answering Ambiguous Questions with Retrieval-Augmented Large Language Models

2023/10/23 by Gangwoo Kim, Sungdong Kim, Kim, Gangwoo +7 · 7 citations
Computer Science · Mathematics · #Ambiguity #Artificial intelligence #Code (set theory) #Computation and Language (cs.CL) #Computer science #Domain (mathematical analysis) #FOS: Computer and information sciences #Information retrieval #Language model #Machine learning #Mathematics #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Natural language processing #One shot #Programming language #Question answering #Set (abstract data type) #Topic Modeling #Tree (set theory)

paper · pdf · doi:10.48550/arxiv.2310.14696

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2023/10/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

Questions in open-domain question answering are often ambiguous, allowing multiple interpretations. One approach to handling them is to identify all possible interpretations of the ambiguous question (AQ) and to generate a long-form answer addressing them all, as suggested by Stelmakh et al., (2022). While it provides a comprehensive response without bothering the user for clarification, considering multiple dimensions of ambiguity and gathering corresponding knowledge remains a challenge. To cope with the challenge, we propose a novel framework, Tree of Clarifications (ToC): It recursively constructs a tree of disambiguations for the AQ -- via few-shot prompting leveraging external knowledge -- and uses it to generate a long-form answer. ToC outperforms existing baselines on ASQA in a few-shot setup across the metrics, while surpassing fully-supervised baselines trained on the whole training set in terms of Disambig-F1 and Disambig-ROUGE. Code is available at https://github.com/gankim/tree-of-clarifications.

Cited by

• Leveraging Spreading Activation for Improved Document Retrieval in Knowledge-Graph-Based RAG Systems
• Learning Steerable Clarification Policies with Collaborative Self-play
• Thinking About Thinking: SAGE-nano's Inverse Reasoning for Self-Aware Language Models
• Deep Research Agents: A Systematic Examination And Roadmap
• REIS: A High-Performance and Energy-Efficient Retrieval System with In-Storage Processing
• Exploring the Landscape of Text-to-SQL with Large Language Models: Progresses, Challenges and Opportunities
• Test-Time Learning for Large Language Models

Related

• Answering Ambiguous Questions with a Database of Questions, Answers, and Revisions
• Asking Clarification Questions to Handle Ambiguity in Open-Domain QA
• Clarifying Ambiguities: on the Role of Ambiguity Types in Prompting Methods for Clarification Generation
• AmbigQA: Answering Ambiguous Open-domain Questions
• CLAM: Selective Clarification for Ambiguous Questions with Generative Language Models