2024/04/19 by Dayton G. Thorpe, Thorpe, Dayton G., Andrew J. Duberstein +3 · 3 citations
Computer Science · Decision Sciences · #Advanced Computational Techniques and Applications #Advanced Database Systems and Queries #Computation and Language (cs.CL) #Databases (cs.DB) #FOS: Computer and information sciences #Scientific Computing and Data Management
paper · pdf · doi:10.48550/arxiv.2404.12560
openalex publication_date 2024/04/19 · openalex created_date 2024/04/23 · openalex updated_date 2026/07/28
The current state-of-the-art (SOTA) for automated text-to-SQL still falls well short of expert human performance as measured by execution accuracy (EX) on the BIRD-SQL benchmark. The most accurate methods are also slow and expensive. To advance the SOTA for text-to-SQL while reducing cost and improving speed, we explore the combination of low-cost fine tuning, novel methods for diverse retrieval-augmented generation (RAG) and new input and output formats that help large language models (LLMs) achieve higher EX. We introduce two new methods, Dubo-SQL v1 and v2. Dubo-SQL v1 sets a new record for EX on the holdout test set of BIRD-SQL. Dubo-SQL v2 achieves even higher performance on the BIRD-SQL dev set. Dubo-SQL v1 relies on LLMs from OpenAI, but uses the low-cost GPT-3.5 Turbo while exceeding the performance of the next-best model using OpenAI, which instead uses the more expensive GPT-4. Dubo-SQL v1 exceeds the performance of the next-best model using GPT-3.5 by over 20%. Dubo-SQL v2 uses GPT-4 Turbo and RAG in place of fine tuning to push EX higher.