CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis

2022/03/25 by Erik Nijkamp, Bo Pang, Nijkamp, Erik +13 · 126 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Programming Languages (cs.PL) #Software Engineering Research #Software Engineering Techniques and Practices #Software Testing and Debugging Techniques

paper · pdf · doi:10.48550/arxiv.2203.13474

openalex publication_date 2022/03/25 · openalex created_date 2022/05/05 · openalex updated_date 2026/07/28

Abstract

Program synthesis strives to generate a computer program as a solution to a given problem specification, expressed with input-output examples or natural language descriptions. The prevalence of large language models advances the state-of-the-art for program synthesis, though limited training resources and data impede open access to such models. To democratize this, we train and release a family of large language models up to 16.1B parameters, called CODEGEN, on natural language and programming language data, and open source the training library JAXFORMER. We show the utility of the trained model by demonstrating that it is competitive with the previous state-of-the-art on zero-shot Python code generation on HumanEval. We further investigate the multi-step paradigm for program synthesis, where a single program is factorized into multiple prompts specifying subproblems. To this end, we construct an open benchmark, Multi-Turn Programming Benchmark (MTPB), consisting of 115 diverse problem sets that are factorized into multi-turn prompts. Our analysis on MTPB shows that the same intent provided to CODEGEN in multi-turn fashion significantly improves program synthesis over that provided as a single turn. We make the training library JAXFORMER and model checkpoints available as open source contribution: https://github.com/salesforce/CodeGen.

Cited by

• Anka: A Domain-Specific Language for Reliable LLM Code Generation
• M2G-Eval: Enhancing and Evaluating Multi-granularity Multilingual Code Generation
• Specification-Driven DevOps for Multi-Service Environments
• Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?
• VSA:Visual-Structural Alignment for UI-to-Code
• Modular Layout Synthesis (MLS): Front-end Code via Structure Normalization and Constrained Generation
• SoK: Understanding (New) Security Issues Across AI4Code Use Cases
• Exploration vs. Fixation: Scaffolding Divergent and Convergent Thinking for Human-AI Co-Creation with Generative Models
• Holistic Evaluation of State-of-the-Art LLMs for Code Generation
• Bridging Natural Language and Formal Specification--Automated Translation of Software Requirements to LTL via Hierarchical Semantics Decomposition Using LLMs
• How Do Semantically Equivalent Code Transformations Impact Membership Inference on LLMs for Code?
• Vector Prism: Animating Vector Graphics by Stratifying Semantic Structure
• Scaling Laws for Code: Every Programming Language Matters
• Towards Privacy-Preserving Code Generation: Differentially Private Code Language Models
• Interpreto: An Explainability Library for Transformers
• Understanding Chain-of-Thought Effectiveness in Code Generation: An Empirical and Information-Theoretic Analysis
• PrivCode: When Code Generation Meets Differential Privacy
• AR-Med: Automated Relevance Enhancement in Medical Search via LLM-Driven Information Augmentation
• Supporting Students in Navigating LLM-Generated Insecure Code
• LLMAID: Identifying AI Capabilities in Android Apps with LLMs
• FAST: Topology-Aware Frequency-Domain Distribution Matching for Coreset Selection
• Cost-Driven Synthesis of Sound Abstract Interpreters
• Go-UT-Bench: A Fine-Tuning Dataset for LLM-Based Unit Test Generation in Go
• ExPairT-LLM: Exact Learning for LLM Code Selection by Pairwise Queries
• A Small Leak Sinks All: Exploring the Transferable Vulnerability of Source Code Models
• From Natural Language to Certified H-infinity Controllers: Integrating LLM Agents with LMI-Based Synthesis
• SemanticForge: Repository-Level Code Generation through Semantic Knowledge Graphs and Constraint Satisfaction
• Optimizing Long-context LLM Serving via Fine-grained Sequence Parallelism
• Self-Abstraction from Grounded Experience for Plan-Guided Policy Refinement
• SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models
• Software Defined Vehicle Code Generation: A Few-Shot Prompting Approach
• From Model to Breach: Towards Actionable LLM-Generated Vulnerabilities Reporting
• PEFA-AI: Advancing Open-source LLMs for RTL generation using Progressive Error Feedback Agentic-AI
• EDIT-Bench: Evaluating LLM Abilities to Perform Real-World Instructed Code Edits
• GEMMA-SQL: A Novel Text-to-SQL Model Based on Large Language Models
• UTF-8 Plumbing: Byte-level Tokenizers Unavoidably Enable LLMs to Generate Ill-formed UTF-8
• ARC-GEN: A Mimetic Procedural Benchmark Generator for the Abstraction and Reasoning Corpus
• EsoLang-Bench: Evaluating Genuine Reasoning in Large Language Models via Esoteric Programming Languages
• Functional Cache Grafting: Robust and Rapid Code-Policy Synthesis for Embodied Agents
• SemOpt: LLM-Driven Code Optimization via Rule-Based Analysis
• Large Language Model for Verilog Code Generation: Literature Review and the Road Ahead
• Agent Data Protocol: Unifying Datasets for Diverse, Effective Fine-tuning of LLM Agents
• Mitigating Hallucination in Large Language Models (LLMs): An Application-Oriented Survey on RAG, Reasoning, and Agentic Systems
• AutoStreamPipe: LLM Assisted Automatic Generation of Data Stream Processing Pipelines
• RefleXGen:The unexamined code is not worth using
• FLAMES: Fine-tuning LLMs to Synthesize Invariants for Smart Contract Security
• R2ComSync: Improving Code-Comment Synchronization with In-Context Learning and Reranking
• Integrating Machine Learning into Belief-Desire-Intention Agents: Current Advances and Open Challenges
• AdaSPEC: Selective Knowledge Distillation for Efficient Speculative Decoders
• AlphaOPT: Formulating Optimization Programs with Self-Improving LLM Experience Library
• Adamas: Hadamard Sparse Attention for Efficient Long-Context Inference
• TREAT: A Code LLMs Trustworthiness / Reliability Evaluation and Testing Framework
• Reasoning Distillation and Structural Alignment for Improved Code Generation
• Breaking Memorization Barriers in LLM Code Fine-Tuning via Information Bottleneck for Improved Generalization
• A Survey on Evaluation of Large Language Models
• Benchmarking Correctness and Security in Multi-Turn Code Generation
• LLMs are All You Need? Improving Fuzz Testing for MOJO with Large Language Models
• Scaling Laws for Code: A More Data-Hungry Regime
• RA-Gen: A Controllable Code Generation Framework Using ReAct for Multi-Agent Task Execution
• Interleaved Learning and Exploration: A Self-Adaptive Fuzz Testing Framework for MLIR
• MOSAIC: Multi-agent Orchestration for Task-Intelligent Scientific Coding
• Fortifying LLM-Based Code Generation with Graph-Based Reasoning on Secure Coding Practices
• Intelligent AI Delegation
• GRACE: A Language Model Framework for Explainable Inverse Reinforcement Learning
• Retrieval-Augmented Code Generation: A Survey with Focus on Repository-Level Approaches
• Refactoring with LLMs: Bridging Human Expertise and Machine Understanding
• Self-Planning Code Generation with Large Language Models
• External Data Extraction Attacks against Retrieval-Augmented Large Language Models
• AutoMaAS: Self-Evolving Multi-Agent Architecture Search for Large Language Models
• AgenticRAG: Tool-Augmented Foundation Models for Zero-Shot Explainable Recommender Systems
• LongCodeZip: Compress Long Context for Code Language Models
• LoRAFusion: Efficient LoRA Fine-Tuning for LLMs
• Latent Thinking Optimization: Your Latent Reasoning Language Model Secretly Encodes Reward Signals in Its Latent Thoughts
• Bridging Developer Instructions and Code Completion Through Instruction-Aware Fill-in-the-Middle Paradigm
• RANGER -- Repository-Level Agent for Graph-Enhanced Retrieval
• The Matthew Effect of AI Programming Assistants: A Hidden Bias in Software Evolution
• Understanding and Enhancing the Planning Capability of Language Models via Multi-Token Prediction
• SR-Eval: Evaluating LLMs on Code Generation under Stepwise Requirement Refinement
• CCrepairBench: A High-Fidelity Benchmark and Reinforcement Learning Framework for C++ Compilation Repair
• LNE-Blocking: An Efficient Framework for Contamination Mitigation Evaluation on Large Language Models
• Evaluating the Effectiveness of Coverage-Guided Fuzzing for Testing Deep Learning Library APIs
• Scrub It Out! Erasing Sensitive Memorization in Code Language Models via Machine Unlearning
• CLMTracing: Black-box User-level Watermarking for Code Language Model Tracing
• Automating Code Generation for Semiconductor Equipment Control from Developer Utterances with LLMs
• HPIM: Heterogeneous Processing-In-Memory-based Accelerator for Large Language Models Inference
• A Systematic Evaluation of Parameter-Efficient Fine-Tuning Methods for the Security of Code LLMs
• Teaching LLMs to Plan: Logical Chain-of-Thought Instruction Tuning for Symbolic Planning
• Large Language Models for Security Operations Centers: A Comprehensive Survey
• LLM-VeriPPA: Power, Performance, and Area Optimization aware Verilog Code Generation with Large Language Models
• AutoVeriFix: Automatically Correcting Errors and Enhancing Functional Correctness in LLM-Generated Verilog Code
• SCoder: Iterative Self-Distillation for Bootstrapping Small-Scale Data Synthesizers to Empower Code LLMs
• ACE and Diverse Generalization via Selective Disagreement
• When Code Crosses Borders: A Security-Centric Study of LLM-based Code Translation
• Tell-Tale Watermarks for Explanatory Reasoning in Synthetic Media Forensics
• Using Contrastive Learning to Improve Two-Way Reasoning in Large Language Models: The Obfuscation Task as a Case Study
• veScale: Consistent and Efficient Tensor Programming with Eager-Mode SPMD
• Symbolic Graphics Programming with Large Language Models
• The Fools are Certain; the Wise are Doubtful: Exploring LLM Confidence in Code Completion
• ASIC-Agent: An Autonomous Multi-Agent System for ASIC Design with Benchmark Evaluation
• Survey of Specialized Large Language Model
• Alignment with Fill-In-the-Middle for Enhancing Code Generation
• CASP: An evaluation dataset for formal verification of C code
• Language Models For Generalised PDDL Planning: Synthesising Sound and Programmatic Policies
• Multi-IaC-Eval: Benchmarking Cloud Infrastructure as Code Across Multiple Formats
• Is GPT-OSS Good? A Comprehensive Evaluation of OpenAI's Latest Open Source Models
• AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunities
• Tabularis Formatus: Predictive Formatting for Tables
• Next Edit Prediction: Learning to Predict Code Edits from Context and Interaction History
• SaraCoder: Orchestrating Semantic and Structural Cues for Resource-Optimized Repository-Level Code Completion
• Optimizing Prompt Sequences using Monte Carlo Tree Search for LLM-Based Optimization
• Bifrost-1: Bridging Multimodal LLMs and Diffusion Models with Patch-level CLIP Latents
• Understanding and Mitigating Errors of LLM-Generated RTL Code
• EvoGraph: Hybrid Directed Graph Evolution toward Software 3.0
• StackPilot: Autonomous Function Agents for Scalable and Environment-Free Code Execution
• Model Compression vs. Adversarial Robustness: An Empirical Study on Language Models for Code
• SAGE-HLS: Syntax-Aware AST-Guided LLM for High-Level Synthesis Code Generation
• BitsAI-Fix: LLM-Driven Approach for Automated Lint Error Resolution in Practice
• NeuroSync: Intent-Aware Code-Based Problem Solving via Direct LLM Understanding Modification
• On the Evaluation of Large Language Models in Multilingual Vulnerability Repair
• TreeRanker: Fast and Model-agnostic Ranking System for Code Suggestions in IDEs
• Prompt to Pwn: Automated Exploit Generation for Smart Contracts
• How Quantization Impacts Privacy Risk on LLMs for Code?
• Enhancing Project-Specific Code Completion by Inferring Internal API Information
• On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey
• The Impact of Fine-tuning Large Language Models on Automated Program Repair
• PurpCode: Reasoning for Safer Code Generation

Related

• Program Synthesis with Large Language Models
• Jigsaw: Large Language Models meet Program Synthesis
• HumanEval on Latest GPT Models -- 2024
• Evaluating Large Language Models Trained on Code
• Function-constrained Program Synthesis