Taming SQL Complexity: LLM-Based Equivalence Evaluation for Text-to-SQL

2025/06/11 by Simin Ma, Zeng, Qingyun, Ma, Simin +6
Computer Science · #Advanced Database Systems and Queries #Natural Language Processing Techniques #Logic, programming, and type systems

paper · pdf · doi:10.48550/arxiv.2506.09359

Abstract

The rise of Large Language Models (LLMs) has significantly advanced Text-to-SQL (NL2SQL) systems, yet evaluating the semantic equivalence of generated SQL remains a challenge, especially given ambiguous user queries and multiple valid SQL interpretations. This paper explores using LLMs to assess both semantic and a more practical "weak" semantic equivalence. We analyze common patterns of SQL equivalence and inequivalence, discuss challenges in LLM-based evaluation.

Citations

• SQL-GEN: Bridging the Dialect Gap for Text-to-SQL Via Synthetic Data And Model Merging
• ETM: Modern Insights into Perspective on Text-to-SQL Evaluation in the Age of Large Language Models
• Self-Tuning: Instructing LLMs to Effectively Acquire New Knowledge through Self-Teaching
• CHESS: Contextual Harnessing for Efficient SQL Synthesis
• SQL-Encoder: Improving NL2SQL In-Context Learning Through a Context-Aware Encoder
• HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decomposition
• LLM-SQL-Solver: Can LLMs Determine SQL Equivalence?
• Calibrating LLM-Based Evaluator
• Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs
• Can Large Language Models Be an Alternative to Human Evaluations?
• DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction
• Hybrid Ranking Network for Text-to-SQL
• Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task

Related

• LLM-SQL-Solver: Can LLMs Determine SQL Equivalence?
• Can the Rookies Cut the Tough Cookie? Exploring the Use of LLMs for SQL Equivalence Checking
• A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going?
• Enhancing LLM Fine-tuning for Text-to-SQLs by SQL Quality Measurement
• Evaluating SQL Understanding in Large Language Models