WaveCoder: Widespread And Versatile Enhancement For Code Large Language Models By Instruction Tuning

2023/12/20 by Zhaojian Yu, Xin Zhang, Yu, Zhaojian +13 · 1 voice · 13 citations
Computer Science · #Software Testing and Debugging Techniques #Software System Performance and Reliability #Parallel Computing and Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2312.14187

Abstract

Recent work demonstrates that, after instruction tuning, Code Large Language Models (Code LLMs) can obtain impressive capabilities to address a wide range of code-related tasks. However, current instruction tuning methods for Code LLMs mainly focus on the traditional code generation task, resulting in poor performance in complex multi-task scenarios. In this paper, we concentrate on multiple code-related tasks and present WaveCoder, a series of Code LLMs trained with Widespread And Versatile Enhanced instruction data. To enable the models to tackle complex code-related tasks, we propose a method to stably generate diverse, high-quality instruction data from open source code dataset in multi-task scenarios and obtain CodeSeaXDataset, a dataset comprising 19,915 instruction instances across 4 code-related tasks, which is aimed at improving the generalization ability of Code LLM. Our experiments demonstrate that WaveCoder models significantly outperform other open-source models in terms of the generalization ability across different code-related tasks. Moreover, WaveCoder-Ultra-6.7B presents the state-of-the-art generalization abilities on a wide range of code-related tasks.

Cited by

• M2G-Eval: Enhancing and Evaluating Multi-granularity Multilingual Code Generation
• CodeSimpleQA: Scaling Factuality in Code Large Language Models
• UCoder: Unsupervised Code Generation by Internal Probing of Large Language Models
• GLaD: Geometric Latent Distillation for Vision-Language-Action Models
• AlphaResearch: Accelerating New Algorithm Discovery with Language Models
• Is Your Prompt Poisoning Code? Defect Induction Rates and Security Mitigation Strategies
• V-GameGym: Visual Game Generation for Code Large Language Models
• Generating High-Quality Datasets for Code Editing via Open-Source Language Models
• Towards Better Correctness and Efficiency in Code Generation
• SCoder: Iterative Self-Distillation for Bootstrapping Small-Scale Data Synthesizers to Empower Code LLMs
• Middo: Model-Informed Dynamic Data Optimization for Enhanced LLM Fine-Tuning via Closed-Loop Learning
• Alignment with Fill-In-the-Middle for Enhancing Code Generation
• CodeEvo: Interaction-Driven Synthesis of Code-centric Data through Hybrid and Iterative Feedback

Discussions

• WaveCoder: Enhanced instruction tuning with refined data generation [hn, 27 points, 10 comments]

Related

• CodeT5+: Open Code Large Language Models for Code Understanding and Generation
• DolphCoder: Echo-Locating Code Large Language Models with Diverse and Multi-Objective Instruction Tuning
• OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs
• Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generation
• WizardCoder: Empowering Code Large Language Models with Evol-Instruct