LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs

2025/04/20 by Xia, Yunhui, Shen, Wei, Wang, Yan +5 · 18 citations
#Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Software Engineering (cs.SE)

paper · doi:10.48550/arxiv.2504.14655

Abstract

We introduce LeetCodeDataset, a high-quality benchmark for evaluating and training code-generation models, addressing two key challenges in LLM research: the lack of reasoning-focused coding benchmarks and self-contained training testbeds. By curating LeetCode Python problems with rich metadata, broad coverage, 100+ test cases per problem, and temporal splits (pre/post July 2024), our dataset enables contamination-free evaluation and efficient supervised fine-tuning (SFT). Experiments show reasoning models significantly outperform non-reasoning counterparts, while SFT with only 2.6K model-generated solutions achieves performance comparable to 110K-sample counterparts. The dataset and evaluation framework are available on Hugging Face and Github.

Cited by

• Holistic Evaluation of State-of-the-Art LLMs for Code Generation
• FrontierCS: Evolving Challenges for Evolving Intelligence
• Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimization
• HarnessLLM: Automatic Testing Harness Generation via Reinforcement Learning
• BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuning
• Expanding LLM Agent Boundaries with Strategy-Guided Exploration
• CodeRL+: Improving Code Generation via Reinforcement with Execution Semantics Alignment
• Enhancing LLM Reasoning via Non-Human-Like Reasoning Path Preference Optimization
• How Many Code and Test Cases Are Enough? Evaluating Test Cases Generation from a Binary-Matrix Perspective
• VeriEquivBench: An Equivalence Score for Ground-Truth-Free Evaluation of Formally Verifiable Code
• Learning on the Job: Test-Time Curricula for Targeted Reinforcement Learning
• Multi-LLM Orchestration for High-Quality Code Generation: Exploiting Complementary Model Strengths
• Don't Settle Too Early: Self-Reflective Remasking for Diffusion Language Models
• Local Success Does Not Compose: Benchmarking Large Language Models for Compositional Formal Verification
• Calibrated Reasoning: An Explanatory Verifier for Dynamic and Efficient Problem-Solving
• From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Users
• Loong: Synthesize Long Chain-of-Thoughts at Scale through Verifiers
• Pruning the Unsurprising: Efficient LLM Reasoning via First-Token Surprisal

Related

• Benchmarking LLMs for Unit Test Generation from Real-World Functions
• CodeLutra: Boosting LLM Code Generation via Preference-Guided Refinement
• Rewriting Pre-Training Data Boosts LLM Performance in Math and Code
• PythonSaga: Redefining the Benchmark to Evaluate Code Generating LLMs
• Prompt engineering and framework: implementation to increase code reliability based guideline for LLMs