Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions

2021/09/13 by Aliannejadi, Mohammad, Kiseleva, Julia, Chuklin, Aleksandr +2 · 4 citations
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Information Retrieval (cs.IR)

paper · doi:10.48550/arxiv.2109.05794

Abstract

Enabling open-domain dialogue systems to ask clarifying questions when appropriate is an important direction for improving the quality of the system response. Namely, for cases when a user request is not specific enough for a conversation system to provide an answer right away, it is desirable to ask a clarifying question to increase the chances of retrieving a satisfying answer. To address the problem of 'asking clarifying questions in open-domain dialogues': (1) we collect and release a new dataset focused on open-domain single- and multi-turn conversations, (2) we benchmark several state-of-the-art neural baselines, and (3) we propose a pipeline consisting of offline and online steps for evaluating the quality of clarifying questions in various dialogues. These contributions are suitable as a foundation for further research.

Cited by

• Sim4IA-Bench: A User Simulation Benchmark Suite for Next Query and Utterance Prediction
• Conversational Search: From Fundamentals to Frontiers in the LLM Era
• TO-GATE: Clarifying Questions and Summarizing Responses with Trajectory Optimization for Eliciting Human Preference
• Disambiguation in Conversational Question Answering in the Era of LLMs and Agents: A Survey

Related

• Deciding Whether to Ask Clarifying Questions in Large-Scale Spoken Language Understanding
• ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ)
• Open-domain clarification question generation without question examples
• Saying No is An Art: Contextualized Fallback Responses for Unanswerable Dialogue Queries
• CLAM: Selective Clarification for Ambiguous Questions with Generative Language Models