Towards Scalable Multi-domain Conversational Agents: The Schema-Guided\n Dialogue Dataset

2019/09/12 by Abhinav Rastogi, Rastogi, Abhinav, Xiaoxue Zang +7 · 49 citations
Computer Science · #AI in Service Interactions #Artificial intelligence #Chatbot #Computer science #Database #Human–computer interaction #Machine learning #Natural language #Natural language understanding #Scalability #Schema (genetic algorithms) #Speech and dialogue systems #Task (project management) #Testbed #Topic Modeling #World Wide Web #cs.CL

paper · pdf · doi:10.48550/arxiv.1909.05855

published in arXiv (Cornell University) (Cornell University) · To appear at AAAI 2020

openalex publication_date 2019/09/12 · arxiv created 2020/01/29 · arxiv updated 2020/01/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Virtual assistants such as Google Assistant, Alexa and Siri provide a\nconversational interface to a large number of services and APIs spanning\nmultiple domains. Such systems need to support an ever-increasing number of\nservices with possibly overlapping functionality. Furthermore, some of these\nservices have little to no training data available. Existing public datasets\nfor task-oriented dialogue do not sufficiently capture these challenges since\nthey cover few domains and assume a single static ontology per domain. In this\nwork, we introduce the the Schema-Guided Dialogue (SGD) dataset, containing\nover 16k multi-domain conversations spanning 16 domains. Our dataset exceeds\nthe existing task-oriented dialogue corpora in scale, while also highlighting\nthe challenges associated with building large-scale virtual assistants. It\nprovides a challenging testbed for a number of tasks including language\nunderstanding, slot filling, dialogue state tracking and response generation.\nAlong the same lines, we present a schema-guided paradigm for task-oriented\ndialogue, in which predictions are made over a dynamic set of intents and\nslots, provided as input, using their natural language descriptions. This\nallows a single dialogue system to easily support a large number of services\nand facilitates simple integration of new services without requiring additional\ntraining data. Building upon the proposed paradigm, we release a model for\ndialogue state tracking capable of zero-shot generalization to new APIs, while\nremaining competitive in the regular setting.\n

Cited by

• ESF-Bench: Benchmarking Challenging Slot-Filling Scenarios for Real-World Enterprise Applications
• CallBench: A Benchmark for Dual-Goal Coordination in Phone Call Assistants
• Beyond Context: Large Language Models' Failure to Grasp Users' Intent
• DIAL: Direct Iterative Adversarial Learning for Realistic Multi-Turn Dialogue Simulation
• Know Your Users! Estimating User Domain Knowledge in Conversational Recommenders
• Faithfulness metric fusion: Improving the evaluation of LLM trustworthiness across domains
• The PLLuM Instruction Corpus
• TOD-ProcBench: Benchmarking Complex Instruction-Following in Task-Oriented Dialogues
• PersonaLens: A Benchmark for Personalization Evaluation in Conversational AI Assistants
• Beyond Task-Oriented and Chitchat Dialogues: Proactive and Transition-Aware Conversational Agents
• Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors
• Thinking While Speaking: Inference-Time Knowledge Transfer for Responsive and Intelligent Conversational Voice Agents
• Living-Harness Is an Interactive-Agent Evolver
• Finding Diamonds in Conversation Haystacks: A Benchmark for Conversational Data Retrieval
• COLA: Continual Learning via Autoencoder Retrieval of Adapters
• Beyond More Context: Retrieval Diversity Boosts Multi-Turn Intent Understanding
• ToolCritic: Detecting and Correcting Tool-Use Errors in Dialogue Systems
• Full-Duplex-Bench-v2: A Multi-Turn Evaluation Framework for Duplex Dialogue Systems with an Automated Examiner
• COMPASS: Benchmarking Constrained Optimization in LLM Agents
• PARSE: LLM Driven Schema Optimization for Reliable Entity Extraction
• DeepJSONEval: Benchmarking Complex Nested JSON Data Mining for Large Language Models
• PARL-MT: Learning to Call Functions in Multi-Turn Conversation with Progress Awareness
• Synthetic Dialogue Generation for Interactive Conversational Elicitation & Recommendation (ICER)
• HiCoLoRA: Addressing Context-Prompt Misalignment via Hierarchical Collaborative LoRA for Zero-Shot DST
• ToolRM: Outcome Reward Models for Tool-Calling Large Language Models
• Quantifying Compositionality of Classic and State-of-the-Art Embeddings
• Multi-Intent Recognition in Dialogue Understanding: A Comparison Between Smaller Open-Source LLMs
• RECAP: REwriting Conversations for Intent Understanding in Agentic Planning
• Schema-Guided Response Generation using Multi-Frame Dialogue State for Motivational Interviewing Systems
• PyTOD: Programmable Task-Oriented Dialogue with Execution Feedback
• MATRIX: Multi-Agent simulaTion fRamework for safe Interactions and conteXtual clinical conversational evaluation
• Open-Universe Assistance Games
• Beyond Solving Math Quiz: Evaluating the Ability of Large Reasoning Models to Ask for Information
• Decoding Neural Emotion Patterns through Large Language Model Embeddings
• Enhancing the Preference Extractor in Multi-turn Dialogues: From Annotating Disasters to Accurate Preference Extraction
• Text-to-SQL Task-oriented Dialogue Ontology Construction
• Goal Alignment in LLM-Based User Simulators for Conversational AI
• DRIVE: Disfluency-Rich Synthetic Dialog Data Generation Framework for Intelligent Vehicle Environments
• Agent WARPP: Workflow Adherence via Runtime Parallel Personalization
• The Behavior Gap: Evaluating Zero-shot LLM Agents in Complex Task-Oriented Dialogs
• Proactive Assistant Dialogue Generation from Streaming Egocentric Videos
• MemGuide: Intent-Driven Memory Selection for Goal-Oriented Multi-Session LLM Agents
• Frictional Agent Alignment Framework: Slow Down and Don't Break Things
• PredAct-Bench: Benchmarking Tool-Augmented Dialogue under Controlled Tool Noise
• Rethinking Stateful Tool Use in Multi-Turn Dialogues: Benchmarks and Challenges
• AURA: A Diagnostic Framework for Tracking User Satisfaction of Interactive Planning Agents
• Schema-Guided Dialogue State Tracking Task at DSTC8
• Generative Induction of Dialogue Task Schemas with Streaming Refinement and Simulated Interactions
• PicPersona-TOD : A Dataset for Personalizing Utterance Style in Task-Oriented Dialogue with Image Persona

Related

• Alexa Conversations: An Extensible Data-driven Approach for Building Task-oriented Dialogue Systems
• Goal-Oriented Multi-Task BERT-Based Dialogue State Tracker
• Manual-Guided Dialogue for Flexible Conversational Agents
• Candidate Attended Dialogue State Tracking Using BERT
• Domain-Aware Dialogue State Tracker for Multi-Domain Dialogue Systems