Can LLMs Narrate Tabular Data? An Evaluation Framework for Natural Language Representations of Text-to-SQL System Outputs

2025/10/27 by Singh, Jyotika, Sun, Weiyi, Agarwal, Amit +4
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2510.23854

Abstract

In modern industry systems like multi-turn chat agents, Text-to-SQL technology bridges natural language (NL) questions and database (DB) querying. The conversion of tabular DB results into NL representations (NLRs) enables the chat-based interaction. Currently, NLR generation is typically handled by large language models (LLMs), but information loss or errors in presenting tabular results in NL remains largely unexplored. This paper introduces a novel evaluation method - Combo-Eval - for judgment of LLM-generated NLRs that combines the benefits of multiple existing methods, optimizing evaluation fidelity and achieving a significant reduction in LLM calls by 25-61%. Accompanying our method is NLR-BIRD, the first dedicated dataset for NLR benchmarking. Through human evaluations, we demonstrate the superior alignment of Combo-Eval with human judgments, applicable across scenarios with and without ground truth references.

Citations

• Aligning LLMs for Multilingual Consistency in Enterprise Applications
• SpeechWeave: Diverse Multilingual Synthetic Text & Audio Data Generation Pipeline for Training Text to Speech Models
• Who's Asking? Investigating Bias Through the Lens of Disability Framed Queries in LLMs
• DAIQ: Auditing Demographic Attribute Inference from Question in LLMs
• Hybrid AI for Responsive Multi-Turn Online Conversations with Novel Dynamic Routing and Feedback Adaptation
• Hard Negative Mining for Domain-Specific Retrieval in Enterprise Systems
• SweEval: Do LLMs Really Swear? A Safety Benchmark for Testing Limits for Enterprise Use
• FS-DAG: Few Shot Domain Adapting Graph Networks for Visually Rich Document Understanding
• Tuning LLM Judge Design Decisions for 1/1000 of the Cost
• Survey of Large Multimodal Model Datasets, Application Categories and Taxonomy
• A Survey on LLM-as-a-Judge
• LLM for Barcodes: Generating Diverse Synthetic Data for Identity Documents
• Text2SQL is Not Enough: Unifying AI and Databases with TAG
• Synthesizing Text-to-SQL Data from Weak and Strong LLMs
• TCSR-SQL: Towards Table Content-aware Text-to-SQL with Self-retrieval
• CHESS: Contextual Harnessing for Efficient SQL Synthesis
• A Survey on Recent Advances in Conversational Data Generation
• Is Table Retrieval a Solved Problem? Exploring Join-Aware Multi-Table Retrieval
• CodeS: Towards Building Open-source Language Models for Text-to-SQL
• Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment
• Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmark
• DBCopilot: Natural Language Querying over Massive Databases via Schema Routing
• SQLPrompt: In-Context Text-to-SQL with Minimal Labeled Data
• BotChat: Evaluating LLMs' Capabilities of Having Multi-Turn Dialogues
• C3: Zero-shot Text-to-SQL with ChatGPT
• Error Detection for Text-to-SQL Semantic Parsing
• Text-to-SQL Error Correction with Language Models of Code
• Interactive Text-to-SQL Generation via Editable Step-by-Step Explanations
• Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs
• DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction
• Augmenting Multi-Turn Text-to-SQL Datasets with Self-Play
• FeTaQA: Free-form Table Question Answering
• ToTTo: A Controlled Table-To-Text Generation Dataset
• BERTScore: Evaluating Text Generation with BERT
• Attention Is All You Need

Related

• From Natural Language to SQL: Review of LLM-based Text-to-SQL Systems
• Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL
• Enhancing LLM Fine-tuning for Text-to-SQLs by SQL Quality Measurement
• Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs
• Natural Language Interfaces for Tabular Data Querying and Visualization: A Survey