CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases

2019/09/11 by Tao Yu, Yu, Tao, Rui Zhang +49 · 34 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech and dialogue systems #Topic Modeling #cs.AI #cs.CL

paper · pdf · doi:10.48550/arxiv.1909.05378

Accepted to EMNLP 2019, long paper

arxiv created 2019/09/11 · openalex publication_date 2019/09/11 · arxiv updated 2019/09/13 · openalex created_date 2022/07/28 · openalex updated_date 2026/07/28

Abstract

We present CoSQL, a corpus for building cross-domain, general-purpose database (DB) querying dialogue systems. It consists of 30k+ turns plus 10k+ annotated SQL queries, obtained from a Wizard-of-Oz (WOZ) collection of 3k dialogues querying 200 complex DBs spanning 138 domains. Each dialogue simulates a real-world DB query scenario with a crowd worker as a user exploring the DB and a SQL expert retrieving answers with SQL, clarifying ambiguous questions, or otherwise informing of unanswerable questions. When user questions are answerable by SQL, the expert describes the SQL and execution results to the user, hence maintaining a natural interaction flow. CoSQL introduces new challenges compared to existing task-oriented dialogue datasets:(1) the dialogue states are grounded in SQL, a domain-independent executable representation, instead of domain-specific slot-value pairs, and (2) because testing is done on unseen databases, success requires generalizing to new domains. CoSQL includes three tasks: SQL-grounded dialogue state tracking, response generation from query results, and user dialogue act prediction. We evaluate a set of strong baselines for each task and show that CoSQL presents significant challenges for future research. The dataset, baselines, and leaderboard will be released at https://yale-lily.github.io/cosql.

Citations

• Expectation-based syntactic comprehension
• On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systems

Cited by

• ABISS: Evaluating Text-to-SQL Systems Through Agent Interaction
• CRAFT: Learn the Schema, Execute the Plan
• An Efficient and Effective Evaluator for Text2SQL Models on Unseen and Unlabeled Data
• FloodSQL-Bench: A Retrieval-Augmented Benchmark for Geospatially-Grounded Text-to-SQL
• Beyond SELECT: A Comprehensive Taxonomy-Guided Benchmark for Real-World Text-to-SQL Translation
• Prompt Engineering Techniques for Context-dependent Text-to-SQL in Arabic
• InData: Towards Secure Multi-Step, Tool-Based Data Analysis
• SCARE: A Benchmark for SQL Correction and Question Answerability Classification for Reliable EHR Question Answering
• SQLSpace: A Representation Space for Text-to-SQL to Discover and Mitigate Robustness Gaps
• Rethinking Text-to-SQL: Dynamic Multi-turn SQL Interaction for Real-world Database Exploration
• A Natural Language Interface for Efficient Data Retrieval in SDSS
• Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questions
• FinAI Data Assistant: LLM-based Financial Database Query Processing with the OpenAI Function Calling API
• MTSQL-R1: Towards Long-Horizon Multi-Turn Text-to-SQL via Agentic Training
• FINCH: Financial Intelligence using Natural language for Contextualized SQL Handling
• BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactions
• Jackal: A Real-World Execution-Based Benchmark Evaluating Large Language Models on Text-to-JQL Tasks
• HyKnow: End-to-End Task-Oriented Dialog Modeling with Hybrid Knowledge Management
• Grounded Adaptation for Zero-shot Executable Semantic Parsing
• Keywords are not always the key: A metadata field analysis for natural language search on open data portals
• Text2Mem: A Unified Memory Operation Language for Memory Operating System
• Castle: Causal Cascade Updates in Relational Databases with Large Language Models
• Decoupled Dialogue Modeling and Semantic Parsing for Multi-Turn Text-to-SQL
• Measuring Data Science Automation: A Survey of Evaluation Tools for AI Assistants and Agents
• Did You Ask a Good Question? A Cross-Domain Question Intention Classification Benchmark for Text-to-SQL
• Toward Real-World Table Agents: Capabilities, Workflows, and Design Principles for LLM-based Table Intelligence
• Improving Text-to-SQL with Schema Dependency Learning
• Offline RL by Reward-Weighted Fine-Tuning for Conversation Optimization
• IGSQL: Database Schema Interaction Graph Based Neural Model for Context-Dependent Text-to-SQL Generation
• Exploring the Landscape of Text-to-SQL with Large Language Models: Progresses, Challenges and Opportunities
• Learning Adaptive Language Interfaces through Decomposition
• RAG Strategies for Natural Language-Based SQL Query and REST API Call Generation
• Evaluating LLMs in Database Scenarios: A Lifecycle Benchmark for Assessing Their Potential in Core Database Tasks
• CLARA: Clarification of Language Ambiguity through Result Analysis for Natural-Language Cancer Genomics Queries
• Abacus-SQL: A Text-to-SQL System Empowering Cross-Domain and Open-Domain Database Retrieval

Related

• Conversational Text-to-SQL: An Odyssey into State-of-the-Art and Challenges Ahead
• Rethinking Text-to-SQL: Dynamic Multi-turn SQL Interaction for Real-world Database Exploration
• OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scale
• STAR: SQL Guided Pre-Training for Context-dependent Text-to-SQL Parsing
• MIGA: A Unified Multi-task Generation Framework for Conversational Text-to-SQL