KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers

2021/06/22 by Chia‐Hsuan Lee, Oleksandr Polozov, Lee, Chia-Hsuan +3 · 13 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Databases (cs.DB) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Programming Languages (cs.PL) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2106.11455

openalex publication_date 2021/06/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The goal of database question answering is to enable natural language querying of real-life relational databases in diverse application domains. Recently, large-scale datasets such as Spider and WikiSQL facilitated novel modeling techniques for text-to-SQL parsing, improving zero-shot generalization to unseen databases. In this work, we examine the challenges that still prevent these techniques from practical deployment. First, we present KaggleDBQA, a new cross-domain evaluation dataset of real Web databases, with domain-specific data types, original formatting, and unrestricted questions. Second, we re-examine the choice of evaluation tasks for text-to-SQL parsers as applied in real-life settings. Finally, we augment our in-domain evaluation task with database documentation, a naturally occurring source of implicit domain knowledge. We show that KaggleDBQA presents a challenge to state-of-the-art zero-shot parsers but a more realistic evaluation setting and creative use of associated database documentation boosts their accuracy by over 13.2%, doubling their performance.

Citations

• Towards Ecologically Valid Research on Language User Interfaces
• GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing
• Semantic Evaluation for Text-to-SQL with Distilled Test Suites

Cited by

• An Efficient and Effective Evaluator for Text2SQL Models on Unseen and Unlabeled Data
• Mistake Notebook Learning: Batch-Clustered Failures for Training-Free Agent Adaptation
• PublicAgent: Multi-Agent Design Principles From an LLM-Based Open Data Analysis Framework
• SQLSpace: A Representation Space for Text-to-SQL to Discover and Mitigate Robustness Gaps
• From Questions to Queries: An AI-powered Multi-Agent Framework for Spatial Text-to-SQL
• Exploring Database Normalization Effects on SQL Generation
• HES-SQL: Hybrid Reasoning for Efficient Text-to-SQL with Structural Skeleton Guidance
• Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and Evaluation
• BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactions
• Retrieval and Augmentation of Domain Knowledge for Text-to-SQL Semantic Parsing
• RubikSQL: Lifelong Learning Agentic Knowledge Base as an Industrial NL2SQL System
• Tabular Data Understanding with LLMs: A Survey of Recent Advances and Challenges
• Text-to-SQL for Enterprise Data Analytics

Related

• "What Do You Mean by That?" A Parser-Independent Interactive Approach for Enhancing Text-to-SQL
• UNITE: A Unified Benchmark for Text-to-SQL Evaluation
• Domain Specific Question to SQL Conversion with Embedded Data Balancing Technique
• Text-to-SQL in the Wild: A Naturally-Occurring Dataset Based on Stack Exchange Data
• CodeS: Towards Building Open-source Language Models for Text-to-SQL