Energy-Based Models for Code Generation under Compilability Constraints

2021/06/09 by Tomasz Korbak, Korbak, Tomasz, Hady Elsahar +5 · 1 voice · 1 citation
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.2 #I.2.6 #I.2.7 #I.5.1 #Machine Learning (cs.LG) #Natural Language Processing Techniques #Neural and Evolutionary Computing (cs.NE) #Software Engineering (cs.SE) #Software Engineering Research #Topic Modeling #cs.CL #cs.LG #cs.NE #cs.SE

paper · pdf · doi:10.48550/arxiv.2106.04985

Accepted for the First Workshop on Natural Language Processing for Programming, ACL 2021

arxiv created 2021/06/09 · openalex publication_date 2021/06/09 · arxiv published 2021/06/09 · arxiv updated 2021/06/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Neural language models can be successfully trained on source code, leading to applications such as code completion. However, their versatile autoregressive self-supervision objective overlooks important global sequence-level features that are present in the data such as syntactic correctness or compilability. In this work, we pose the problem of learning to generate compilable code as constraint satisfaction. We define an Energy-Based Model (EBM) representing a pre-trained generative model with an imposed constraint of generating only compilable sequences. We then use the KL-Adaptive Distributional Policy Gradient algorithm (Khalifa et al., 2021) to train a generative model approximating the EBM. We conduct experiments showing that our proposed approach is able to improve compilability rates without sacrificing diversity and complexity of the generated samples.

Citations

• Adam: A Method for Stochastic Optimization
• Training Products of Experts by Minimizing Contrastive Divergence
• A Deep Reinforced Model for Abstractive Summarization
• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Fine-Tuning Language Models from Human Preferences
• Distributional Reinforcement Learning for Energy-Based Sequential Models
• ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translation
• Unsupervised Translation of Programming Languages
• CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation

Cited by

• Incorporating Inductive Biases to Energy-based Generative Models

Discussions

• Energy-Based Models for Code Generation Under Compilability Constraints [hn, 1 points, 0 comments]

Related

• A Distributional Approach to Controlled Text Generation
• Compilable Neural Code Generation with Compiler Feedback
• Controlling Conditional Language Models without Catastrophic Forgetting
• Automatic Code Generation using Pre-Trained Language Models
• Leveraging Grammar and Reinforcement Learning for Neural Program Synthesis