DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation
2022/11/18 by Yuhang Lai, Chengxi Li, Lai, Yuhang +17 · 75 citations
Computer Science · Materials Science · #Software Engineering Research #Machine Learning and Data Classification #Machine Learning in Materials Science
paper · pdf · doi:10.48550/arxiv.2211.11501
Abstract
We introduce DS-1000, a code generation benchmark with a thousand data science problems spanning seven Python libraries, such as NumPy and Pandas. Compared to prior works, DS-1000 incorporates three core features. First, our problems reflect diverse, realistic, and practical use cases since we collected them from StackOverflow. Second, our automatic evaluation is highly specific (reliable) -- across all Codex-002-predicted solutions that our evaluation accept, only 1.8% of them are incorrect; we achieve this with multi-criteria metrics, checking both functional correctness by running test cases and surface-form constraints by restricting API usages or keywords. Finally, we proactively defend against memorization by slightly modifying our problems to be different from the original StackOverflow source; consequently, models cannot answer them correctly by memorizing the solutions from pre-training. The current best public system (Codex-002) achieves 43.3% accuracy, leaving ample room for improvement. We release our benchmark at https://ds1000-code-gen.github.io.
Cited by
- Java academic benchmark: Exam-based evaluation of LLMs on object-oriented programming
- Many AI Analysts, One Dataset: Navigating the Agentic Data Science Multiverse
- HiSciBench: A Hierarchical Multi-disciplinary Benchmark for Scientific Intelligence from Reading to Discovery
- FEM-Bench: A Structured Scientific Reasoning Benchmark for Evaluating Code-Generating LLMs
- SoK: Understanding (New) Security Issues Across AI4Code Use Cases
- CIFE: Code Instruction-Following Evaluation
- Bolmo: Byteifying the Next Generation of Language Models
- Market-Bench: Evaluating Large Language Models on Introductory Quantitative Trading and Market Dynamics
- Confucius Code Agent: Scalable Agent Scaffolding for Real-World Codebases
- JT-DA: Enhancing Data Analysis with Tool-Integrated Table Reasoning Large Language Models
- DataGovBench: Benchmarking LLM Agents for Real-World Data Governance Workflows
- DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle
- CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Coding
- InsightEval: An Expert-Curated Benchmark for Assessing Insight Discovery in LLM-Driven Data Agents
- AttackPilot: Autonomous Inference Attacks Against ML Services With LLM-Based Agents
- UniDataBench: Evaluating Data Analytics Agents Across Structured and Unstructured Data
- InData: Towards Secure Multi-Step, Tool-Based Data Analysis
- ExPairT-LLM: Exact Learning for LLM Code Selection by Pairwise Queries
- Smarter Together: Creating Agentic Communities of Practice through Shared Experiential Learning
- A Hierarchical Imprecise Probability Approach to Reliability Assessment of Large Language Models
- QCoder Benchmark: Bridging Language Generation and Quantum Hardware through Simulator-Based Feedback
- OrchBench: Evaluating Multi-Agent Orchestration Plans in Isolation via Deterministic Simulation
- UrbanDS: A Graph-Guided LLM Multi-Agent System for Data-Intensive Urban Tasks
- ClinLens: Towards Long-Horizon Coding Agents for Longitudinal Multimodal Clinical Data Science
- Cross-Model Cross-Language AI Coding Agent Performance: Accuracy and Speed of Parallel CLRS Algorithms
- EsoLang-Bench: Evaluating Genuine Reasoning in Large Language Models via Esoteric Programming Languages
- Exploiting LLM Agent Supply Chains via Payload-less Skills
- A Survey on LLM Mid-Training
- A Survey of AI Scientists
- DeepAnalyze: Agentic Large Language Models for Autonomous Data Science
- QuanBench: Benchmarking Quantum Code Generation with Large Language Models
- Faster LLM Inference via Sequential Monte Carlo
- LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?
- Identifying & Interactively Refining Ambiguous User Goals for Data Visualization Code Generation
- Metric Calculating Benchmark: Code-Verifiable Complicate Instruction Following Benchmark for Large Language Models
- LLM-Based Multi-Agent Blackboard System for Information Discovery in Data Science
- ML2B: Benchmarking LLMs on Cross-Lingual ML Pipeline Generation
- DataClawEval: A Benchmark for Data Engineering Agents in Real Industrial Harness
- SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvements
- Large Language Models for Accessible Reporting of Bioinformatics Analyses in Interdisciplinary Contexts
- LIMI: Less is More for Agency
- CFDLLMBench: A Benchmark Suite for Evaluating Large Language Models in Computational Fluid Dynamics
- VisDocSketcher: Towards Scalable Visual Documentation with Agentic Systems
- SciML Agents: Write the Solver, Not the Solution
- ImportSnare: Directed "Code Manual" Hijacking in Retrieval-Augmented Code Generation
- Another Turn, Better Output? A Turn-Wise Analysis of Iterative LLM Prompting
- FDABench: A Benchmark for Data Agents on Analytical Queries over Heterogeneous Data
- 11Plus-Bench: Demystifying Multimodal LLM Spatial Reasoning with Cognitive-Inspired Analysis
- ReST-RL: Achieving Accurate Code Reasoning of LLMs with Optimized Self-Training and Decoding
- GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging
- COMPASS: A Multi-Dimensional Benchmark for Evaluating Code Generation in Large Language Models
- Large Language Models in the Data Science Lifecycle: A Systematic Mapping Study
- Non-programmers Assessing AI-Generated Code: A Case Study of Business Users Analyzing Data
- More Than a Score: Probing the Impact of Prompt Specificity on LLM Code Generation
- Large Language Model-based Data Science Agent: A Survey
- AutoBridge: Automating Smart Device Integration with Centralized Platform
- DSBC : Data Science task Benchmarking with Context engineering
- IFEvalCode: Controlled Code Generation
- Persona-Augmented Benchmarking: Evaluating LLMs Across Diverse Writing Styles
- ASPERA: A Simulated Environment to Evaluate Planning for Complex Action Execution
- When Prompts Go Wrong: Evaluating Code Model Robustness to Ambiguous, Contradictory, and Incomplete Task Descriptions
- CodegenBench: Can LLMs Write Efficient Code Across Architectures?
- Business Utility of Large Language Models as Exploratory Data Analysis Agents
- LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces
- Toward Real-World Table Agents: Capabilities, Workflows, and Design Principles for LLM-based Table Intelligence
- Evaluating LLMs on Sequential API Call Through Automated Test Generation
- MateInfoUB: A Real-World Benchmark for Testing LLMs in Competitive, Multilingual, and Multimodal Educational Tasks
- A Hierarchical and Evolvable Benchmark for Fine-Grained Code Instruction Following with Multi-Turn Feedback
- A Survey of LLM-based Automated Program Repair: Taxonomies, Design Paradigms, and Applications
- DABstep: Data Agent Benchmark for Multi-step Reasoning
- Training Language Model to Critique for Better Refinement
- ReCode: Updating Code API Knowledge with Reinforcement Learning
- SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applications
- OJBench: A Competition Level Code Benchmark For Large Language Models
- Humanity's Last Code Exam: Can Advanced LLMs Conquer Human's Hardest Code Competition?
Related