CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain\n Natural Language Interfaces to Databases

2019/09/11 by Tao Yu, Rui Zhang, Yu, Tao +47 · 24 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech and dialogue systems #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1909.05378

openalex publication_date 2019/09/11 · openalex created_date 2022/07/28 · openalex updated_date 2026/07/28

Abstract

We present CoSQL, a corpus for building cross-domain, general-purpose\ndatabase (DB) querying dialogue systems. It consists of 30k+ turns plus 10k+\nannotated SQL queries, obtained from a Wizard-of-Oz (WOZ) collection of 3k\ndialogues querying 200 complex DBs spanning 138 domains. Each dialogue\nsimulates a real-world DB query scenario with a crowd worker as a user\nexploring the DB and a SQL expert retrieving answers with SQL, clarifying\nambiguous questions, or otherwise informing of unanswerable questions. When\nuser questions are answerable by SQL, the expert describes the SQL and\nexecution results to the user, hence maintaining a natural interaction flow.\nCoSQL introduces new challenges compared to existing task-oriented dialogue\ndatasets:(1) the dialogue states are grounded in SQL, a domain-independent\nexecutable representation, instead of domain-specific slot-value pairs, and (2)\nbecause testing is done on unseen databases, success requires generalizing to\nnew domains. CoSQL includes three tasks: SQL-grounded dialogue state tracking,\nresponse generation from query results, and user dialogue act prediction. We\nevaluate a set of strong baselines for each task and show that CoSQL presents\nsignificant challenges for future research. The dataset, baselines, and\nleaderboard will be released at https://yale-lily.github.io/cosql.\n

Citations

• Expectation-based syntactic comprehension
• On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systems

Cited by

• ABISS: Evaluating Text-to-SQL Systems Through Agent Interaction
• CRAFT: Learn the Schema, Execute the Plan
• An Efficient and Effective Evaluator for Text2SQL Models on Unseen and Unlabeled Data
• FloodSQL-Bench: A Retrieval-Augmented Benchmark for Geospatially-Grounded Text-to-SQL
• Beyond SELECT: A Comprehensive Taxonomy-Guided Benchmark for Real-World Text-to-SQL Translation
• Prompt Engineering Techniques for Context-dependent Text-to-SQL in Arabic
• InData: Towards Secure Multi-Step, Tool-Based Data Analysis
• SCARE: A Benchmark for SQL Correction and Question Answerability Classification for Reliable EHR Question Answering
• SQLSpace: A Representation Space for Text-to-SQL to Discover and Mitigate Robustness Gaps
• Rethinking Text-to-SQL: Dynamic Multi-turn SQL Interaction for Real-world Database Exploration
• A Natural Language Interface for Efficient Data Retrieval in SDSS
• Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questions
• FinAI Data Assistant: LLM-based Financial Database Query Processing with the OpenAI Function Calling API
• MTSQL-R1: Towards Long-Horizon Multi-Turn Text-to-SQL via Agentic Training
• FINCH: Financial Intelligence using Natural language for Contextualized SQL Handling
• BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactions
• Jackal: A Real-World Execution-Based Benchmark Evaluating Large Language Models on Text-to-JQL Tasks
• HyKnow: End-to-End Task-Oriented Dialog Modeling with Hybrid Knowledge Management
• Grounded Adaptation for Zero-shot Executable Semantic Parsing
• Keywords are not always the key: A metadata field analysis for natural language search on open data portals
• Text2Mem: A Unified Memory Operation Language for Memory Operating System
• Castle: Causal Cascade Updates in Relational Databases with Large Language Models
• Decoupled Dialogue Modeling and Semantic Parsing for Multi-Turn Text-to-SQL
• Did You Ask a Good Question? A Cross-Domain Question Intention Classification Benchmark for Text-to-SQL
• Toward Real-World Table Agents: Capabilities, Workflows, and Design Principles for LLM-based Table Intelligence

Related

• Conversational Text-to-SQL: An Odyssey into State-of-the-Art and Challenges Ahead
• Rethinking Text-to-SQL: Dynamic Multi-turn SQL Interaction for Real-world Database Exploration
• OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scale
• STAR: SQL Guided Pre-Training for Context-dependent Text-to-SQL Parsing
• MIGA: A Unified Multi-task Generation Framework for Conversational Text-to-SQL