OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs

2025/04/05 by Wasi Uddin Ahmad, Aleksander Ficek, Ahmad, Wasi Uddin +11 · 9 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Software Engineering (cs.SE) #Software Engineering Research #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2504.04030

openalex publication_date 2025/04/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) have transformed software development by enabling code generation, automated debugging, and complex reasoning. However, their continued advancement is constrained by the scarcity of high-quality, publicly available supervised fine-tuning (SFT) datasets tailored for coding tasks. To bridge this gap, we introduce OpenCodeInstruct, the largest open-access instruction tuning dataset, comprising 5 million diverse samples. Each sample includes a programming question, solution, test cases, execution feedback, and LLM-generated quality assessments. We fine-tune various base models, including LLaMA and Qwen, across multiple scales (1B+, 3B+, and 7B+) using our dataset. Comprehensive evaluations on popular benchmarks (HumanEval, MBPP, LiveCodeBench, and BigCodeBench) demonstrate substantial performance improvements achieved by SFT with OpenCodeInstruct. We also present a detailed methodology encompassing seed data curation, synthetic instruction and solution generation, and filtering.

Cited by

• Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
• Toward Explaining Large Language Models in Software Engineering Tasks
• DEER: Draft with Diffusion, Verify with Autoregressive Models
• Fast and Accurate Causal Parallel Decoding using Jacobi Forcing
• From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models
• PRO: Enabling Precise and Robust Text Watermark for Open-Source LLMs
• WebGen-V Bench: Structured Representation for Enhancing Visual Design in LLM-based Web Generation and Evaluation
• NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model
• Your Coding Intent is Secretly in the Context and You Should Deliberately Infer It Before Completion

Related

• How Do Your Code LLMs Perform? Empowering Code Instruction Tuning with High-Quality Data
• Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generation
• Evaluating and Aligning CodeLLMs on Human Preference
• OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models
• CodecLM: Aligning Language Models with Tailored Synthetic Data