InCoder: A Generative Model for Code Infilling and Synthesis

2022/04/12 by Daniel Fried, Fried, Daniel, Armen Aghajanyan +16 · 142 citations
Computer Science · #Artificial intelligence #Code (set theory) #Code generation #Computer science #Context (archaeology) #Generative grammar #Generative model #Inference #Key (lock) #Programming language #Set (abstract data type) #Software Engineering Research #Software Reliability and Analysis Research #Software Testing and Debugging Techniques

paper · pdf · doi:10.48550/arxiv.2204.05999

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2022/04/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Code is seldom written in a single left-to-right pass and is instead repeatedly edited and refined. We introduce InCoder, a unified generative model that can perform program synthesis (via left-to-right generation) as well as editing (via infilling). InCoder is trained to generate code files from a large corpus of permissively licensed code, where regions of code have been randomly masked and moved to the end of each file, allowing code infilling with bidirectional context. Our model is the first generative model that is able to directly perform zero-shot code infilling, which we evaluate on challenging tasks such as type inference, comment generation, and variable re-naming. We find that the ability to condition on bidirectional context substantially improves performance on these tasks, while still performing comparably on standard program synthesis benchmarks in comparison to left-to-right only models pretrained at similar scale. The InCoder models and code are publicly released. https://sites.google.com/view/incoder-code-models

Cited by

• Learning from 53.6K Real-World Developer Edits of AI-Generated Code
• Exploring the Security Threats of Retriever Backdoors in Retrieval-Augmented Code Generation
• Co-Evolution of Types and Dependencies: Towards Repository-Level Type Inference for Python Code
• VSA:Visual-Structural Alignment for UI-to-Code
• Modular Layout Synthesis (MLS): Front-end Code via Structure Normalization and Constrained Generation
• Attention Distance: A Novel Metric for Directed Fuzzing with Large Language Models
• Decoding Human-LLM Collaboration in Coding: An Empirical Study of Multi-Turn Conversations in the Wild
• Understanding Chain-of-Thought Effectiveness in Code Generation: An Empirical and Information-Theoretic Analysis
• PrivCode: When Code Generation Meets Differential Privacy
• LLMAID: Identifying AI Capabilities in Android Apps with LLMs
• GenSIaC: Toward Security-Aware Infrastructure-as-Code Generation with Large Language Models
• FLEX: Continuous Agent Evolution via Forward Learning from Experience
• SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models
• EDIT-Bench: Evaluating LLM Abilities to Perform Real-World Instructed Code Edits
• Context-Guided Decompilation: A Step Towards Re-executability
• DPO-F+: Aligning Code Repair Feedback with Developers' Preferences
• ARC-GEN: A Mimetic Procedural Benchmark Generator for the Abstraction and Reasoning Corpus
• EsoLang-Bench: Evaluating Genuine Reasoning in Large Language Models via Esoteric Programming Languages
• MATCH: Task-Driven Code Evaluation through Contrastive Learning
• R2ComSync: Improving Code-Comment Synchronization with In-Context Learning and Reranking
• TREAT: A Code LLMs Trustworthiness / Reliability Evaluation and Testing Framework
• Reasoning Distillation and Structural Alignment for Improved Code Generation
• Beyond Function-Level Search: Repository-Aware Dual-Encoder Code Retrieval with Adversarial Verification
• Unlocking the Potential of Diffusion Language Models through Template Infilling
• Testing and Enhancing Multi-Agent Systems for Robust Code Generation
• LLMs are All You Need? Improving Fuzz Testing for MOJO with Large Language Models
• Towards Speeding up Program Repair with Non-Autoregressive Model
• Scaling Laws for Code: A More Data-Hungry Regime
• RA-Gen: A Controllable Code Generation Framework Using ReAct for Multi-Agent Task Execution
• Fortifying LLM-Based Code Generation with Graph-Based Reasoning on Secure Coding Practices
• GRACE: A Language Model Framework for Explainable Inverse Reinforcement Learning
• AutoMaAS: Self-Evolving Multi-Agent Architecture Search for Large Language Models
• Breaking the Code: Security Assessment of AI Code Agents Through Systematic Jailbreaking Attacks
• Combining Large Language Models and Gradient-Free Optimization for Automatic Control Policy Synthesis
• Bridging Developer Instructions and Code Completion Through Instruction-Aware Fill-in-the-Middle Paradigm
• MultiMat: Multimodal Program Synthesis for Procedural Materials using Large Multimodal Models
• SCOPE: Synthetic Conditional Objectives for Policy Evolution in Black-Box Combinatorial Optimization
• LLaVul: A Multimodal LLM for Interpretable Vulnerability Reasoning about Source Code
• Generating High-Quality Datasets for Code Editing via Open-Source Language Models
• Evaluating the Effectiveness of Coverage-Guided Fuzzing for Testing Deep Learning Library APIs
• Scrub It Out! Erasing Sensitive Memorization in Code Language Models via Machine Unlearning
• Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues
• UserTrace: User-Level Requirements Generation and Traceability Recovery from Software Project Repositories
• ReCode: Improving LLM-based Code Repair with Fine-Grained Retrieval-Augmented Generation
• RepoMark: A Data-Usage Auditing Framework for Code Large Language Models
• Alignment with Fill-In-the-Middle for Enhancing Code Generation
• Is GPT-OSS Good? A Comprehensive Evaluation of OpenAI's Latest Open Source Models
• Constrained Decoding of Diffusion LLMs with Context-Free Grammars
• Next Edit Prediction: Learning to Predict Code Edits from Context and Interaction History
• What Builds Effective In-Context Examples for Code Generation?
• TRAIL: Joint Inference and Refinement of Knowledge Graphs with Large Language Models
• DRAMA: A Dynamic and Robust Allocation-based Multi-Agent System for Changing Environments
• StackPilot: Autonomous Function Agents for Scalable and Environment-Free Code Execution
• LaTCoder: Converting Webpage Design to Code with Layout-as-Thought
• MRG-Bench: Evaluating and Exploring the Requirements of Context for Repository-Level Code Generation
• SMART-Editor: A Multi-Agent Framework for Human-Like Design Editing with Structural Integrity
• AlgoTune: Can Language Models Speed Up General-Purpose Numerical Programs?
• Kodezi Chronos: A Debugging-First Language Model for Repository-Scale Code Understanding
• Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction
• RefineX: Learning to Refine Pre-training Data at Scale from Expert-Guided Programs
• LEDOM: Reverse Language Model
• PSearch: Search-based Patch Generation in the Era of LLM-based Automated Program Repair
• Recommending Variable Names for Extract Local Variable Refactorings
• Failure by Interference: Language Models Make Balanced Parentheses Errors When Faulty Mechanisms Overshadow Sound Ones
• A Survey of LLM-based Automated Program Repair: Taxonomies, Design Paradigms, and Applications
• QLPro: Automated Code Vulnerability Discovery via LLM and Static Code Analysis Integration
• Adversarial Attack Classification and Robustness Testing for Large Language Models for Code
• Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation
• T³: Multi-level Tree-based Automatic Program Repair with Large Language Models
• From Empirical Evaluation to Context-Aware Enhancement: Repairing Regression Errors with LLMs
• SCGAgent: Recreating the Benefits of Reasoning Models for Secure Code Generation with Agentic Workflows
• Tokenized Bandit for LLM Decoding and Alignment
• Seed-Coder: Let the Code Model Curate Data for Itself
• A Preference-Driven Methodology for High-Quality Solidity Code Generation
• CoQuIR: A Comprehensive Benchmark for Code Quality-Aware Information Retrieval
• Structure-Aware Fill-in-the-Middle Pretraining for Code
• EFIM: Efficient Serving of LLMs for Infilling Tasks with Improved KV Cache Reuse
• From Output to Evaluation: Does Raw Instruction-Tuned Code LLMs Output Suffice for Fill-in-the-Middle Code Generation?
• ECLAIR: A Causally-Grounded AI Framework for Scientific Discovery in Empirical Software Engineering
• EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code
• Unreliable in Practice? A Comprehensive Study of Errors in LLM-Generated Code
• SuperCoder: Assembly Program Superoptimization with Large Language Models
• VeriThoughts: Enabling Automated Verilog Code Generation using Reasoning and Formal Verification
• UICopilot: Automating UI Synthesis via Hierarchical Code Generation from Webpage Designs
• Enhancing Code Generation via Bidirectional Comment-Level Mutual Grounding
• Benchmarking and Revisiting Code Generation Assessment: A Mutation-Based Approach
• Syntax Without Semantics: Teaching Large Language Models to Code in an Unseen Language
• Evolutionary thoughts: integration of large language models and evolutionary algorithms
• Towards Mitigating API Hallucination in Code Generated by LLMs with Hierarchical Dependency Aware
• A Path Less Traveled: Reimagining Software Engineering Automation via a Neurosymbolic Paradigm
• Frontier Coding Agents Use Metaprogramming to Adapt to Unfamiliar Programming Languages
• Composer 2 Technical Report
• An Empirical Study on the Effectiveness of Large Language Models for Binary Code Understanding
• Reviving Any-Subset Autoregressive Models with Principled Parallel Sampling and Speculative Decoding
• A Systematic Literature Review of Parameter-Efficient Fine-Tuning for Large Code Models
• A Catalog of Data Smells for Coding Tasks
• A Multi-Language Perspective on the Robustness of LLM Code Generation
• The Order Is the Guarantee: Verifier-Budgeted Code Deletion with Static-First Learned Proposals
• AdaCoder: An Adaptive Planning and Multi-Agent Framework for Function-Level Code Generation
• MSCoT: Structured Chain-of-Thought Generation for Multiple Programming Languages
• Migrating Code At Scale With LLMs At Google
• MigGPT: Harnessing Large Language Models for Automated Migration of Out-of-Tree Linux Kernel Patches Across Versions
• LSR-MCTS: Alleviating Long Range Dependency in Code Generation
• Evaluating Small-Scale Code Models for Code Clone Detection
• SciSciGPT: Advancing Human-AI Collaboration in the Science of Science

Related

• Self-Infilling Code Generation
• EgoCoder: Intelligent Program Synthesis with Hierarchical Sequential Neural Network Model
• EpiCoder: Encompassing Diversity and Complexity in Code Generation
• ExeDec: Execution Decomposition for Compositional Generalization in Neural Program Synthesis
• NatGen: Generative pre-training by "Naturalizing" source code