Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows
2024/11/12 by Fangyu Lei, Jixuan Chen, Lei, Fangyu +29 · 66 citations
Business, Management and Accounting · Computer Science · Decision Sciences · #Business Process Modeling and Analysis #Service-Oriented Architecture and Web Services #Scientific Computing and Data Management
paper · pdf · doi:10.48550/arxiv.2411.07763
Abstract
Real-world enterprise text-to-SQL workflows often involve complex cloud or local data across various database systems, multiple SQL queries in various dialects, and diverse operations from data transformation to analytics. We introduce Spider 2.0, an evaluation framework comprising 632 real-world text-to-SQL workflow problems derived from enterprise-level database use cases. The databases in Spider 2.0 are sourced from real data applications, often containing over 1,000 columns and stored in local or cloud database systems such as BigQuery and Snowflake. We show that solving problems in Spider 2.0 frequently requires understanding and searching through database metadata, dialect documentation, and even project-level codebases. This challenge calls for models to interact with complex SQL workflow environments, process extremely long contexts, perform intricate reasoning, and generate multiple SQL queries with diverse operations, often exceeding 100 lines, which goes far beyond traditional text-to-SQL challenges. Our evaluations indicate that based on o1-preview, our code agent framework successfully solves only 21.3% of the tasks, compared with 91.2% on Spider 1.0 and 73.0% on BIRD. Our results on Spider 2.0 show that while language models have demonstrated remarkable performance in code generation -- especially in prior text-to-SQL benchmarks -- they require significant improvement in order to achieve adequate performance for real-world enterprise usage. Progress on Spider 2.0 represents crucial steps towards developing intelligent, autonomous, code agents for real-world enterprise settings. Our code, baseline models, and data are available at https://spider2-sql.github.io
Cited by
- Bridging Global Intent with Local Details: A Hierarchical Representation Approach for Semantic Validation in Text-to-SQL
- Cost-Aware Text-to-SQL: An Empirical Study of Cloud Compute Costs for LLM-Generated Queries
- From Unstructured Recall to Schema-Grounded Memory: Reliable AI Memory via Iterative, Schema-Aware Extraction
- CRAFT: Learn the Schema, Execute the Plan
- Learning When to Reason for Text-to-SQL via SFT and DPO
- An Efficient and Effective Evaluator for Text2SQL Models on Unseen and Unlabeled Data
- CricBench: A Multilingual Benchmark for Evaluating LLMs in Cricket Analytics
- From Natural Language to Control Signals: A Conceptual Framework for Semantic Channel Finding in Complex Experimental Infrastructure
- Scaling Text2SQL via LLM-efficient Schema Filtering with Functional Dependency Graph Rerankers
- MedInsightBench: Evaluating Medical Analytics Agents Through Multi-Step Insight Discovery in Multimodal Medical Data
- Exqutor: Extended Query Optimizer for Vector-augmented Analytical Queries
- Can AI autonomously build, operate, and use the entire data stack?
- RLAX: Large-Scale, Distributed Reinforcement Learning for Large Language Models on TPUs
- DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle
- Evidence-Guided Schema Normalization for Temporal Tabular Reasoning
- InsightEval: An Expert-Curated Benchmark for Assessing Insight Discovery in LLM-Driven Data Agents
- Text-to-SQL as Dual-State Reasoning: Integrating Adaptive Context and Progressive Generation
- LLM and Agent-Driven Data Analysis: A Systematic Approach for Enterprise Applications and System-level Deployment
- AskDB: An LLM Agent for Natural Language Interaction with Relational Databases
- Beyond SELECT: A Comprehensive Taxonomy-Guided Benchmark for Real-World Text-to-SQL Translation
- UniDataBench: Evaluating Data Analytics Agents Across Structured and Unstructured Data
- SynQuE: Estimating Synthetic Dataset Quality Without Annotations
- Test-Time Adaptation for LLM Agents via Environment Interaction
- MARS-SQL: A multi-agent reinforcement learning framework for Text-to-SQL
- Can LLMs Help You at Work? A Sandbox for Evaluating LLM Agents in Enterprise Environments
- DRAMA: Unifying Data Retrieval and Analysis for Open-Domain Analytic Queries
- SQLSpace: A Representation Space for Text-to-SQL to Discover and Mitigate Robustness Gaps
- Rethinking Text-to-SQL: Dynamic Multi-turn SQL Interaction for Real-world Database Exploration
- SpotIt: Evaluating Text-to-SQL Evaluation with Formal Verification
- Declarative Techniques for NL Queries over Heterogeneous Data
- Falcon: A Comprehensive Chinese Text-to-SQL Benchmark for Enterprise-Grade Evaluation
- DynaQuery: A Self-Adapting Framework for Querying Structured and Multimodal Data
- Enterprise Deep Research: Steerable Multi-Agent Deep Research for Enterprise Analytics
- Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agents
- JudgeSQL: Reasoning over SQL Candidates with Weighted Consensus Tournament
- Agentic NL2SQL to Reduce Computational Costs
- ConsintBench: Evaluating Language Models on Real-World Consumer Intent Understanding
- Exploring Database Normalization Effects on SQL Generation
- Role-Conditioned Refusals: Evaluating Access Control Reasoning in Large Language Models
- BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactions
- EMR-AGENT: Automating Cohort and Feature Extraction from EMR Databases
- Retrieval and Augmentation of Domain Knowledge for Text-to-SQL Semantic Parsing
- Thinkquel: A Model Dedicated to Text-to-dbt Using Synthetic Data and a Span-Aware Objective
- Scaling Generalist Data-Analytic Agents
- Multilingual Text-to-SQL: Benchmarking the Limits of Language Models with Collaborative Language Agents
- Jackal: A Real-World Execution-Based Benchmark Evaluating Large Language Models on Text-to-JQL Tasks
- MaskSQL: Safeguarding Privacy for LLM-Based Text-to-SQL via Abstraction
- Same Content, Different Representations: A Controlled Study for Table QA
- RL Grokking Recipe: How Does RL Unlock and Transfer New Algorithms in LLMs?
- SteinerSQL: Graph-Guided Mathematical Reasoning for Text-to-SQL Generation
- Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agents
- DataClawEval: A Benchmark for Data Engineering Agents in Real Industrial Harness
- Open Security Benchmark: Towards Autonomous Enterprise Cyber Defense
- Propose and Rectify: A Forensics-Driven MLLM Framework for Image Manipulation Localization
- RubikSQL: Lifelong Learning Agentic Knowledge Base as an Industrial NL2SQL System
- SQLGovernor: An LLM-powered SQL Toolkit for Real World Application
- PaVeRL-SQL: Text-to-SQL via Partial-Match Rewards and Verbal Reinforcement Learning
- Evaluating NL2SQL via SQL2NL
- FDABench: A Benchmark for Data Agents on Analytical Queries over Heterogeneous Data
- AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunities
- WideSearch: Benchmarking Agentic Broad Info-Seeking
- Data-Aware Socratic Query Refinement in Database Systems
- WebDS: An End-to-End Benchmark for Web-based Data Science
- Tabular Data Understanding with LLMs: A Survey of Recent Advances and Challenges
- VeriMinder: Mitigating Analytical Vulnerabilities in NL2SQL
- Text-to-SQL for Enterprise Data Analytics
Related