Unified Pre-training for Program Understanding and Generation

2021/03/10 by Wasi Uddin Ahmad, Ahmad, Wasi Uddin, Saikat Chakraborty +5 · 26 citations
Computer Science · #Advanced Malware Detection Techniques #Computation and Language (cs.CL) #FOS: Computer and information sciences #Programming Languages (cs.PL) #Software Engineering Research #Software Testing and Debugging Techniques

paper · pdf · doi:10.48550/arxiv.2103.06333

openalex publication_date 2021/03/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Code summarization and generation empower conversion between programming language (PL) and natural language (NL), while code translation avails the migration of legacy code from one PL to another. This paper introduces PLBART, a sequence-to-sequence model capable of performing a broad spectrum of program and language understanding and generation tasks. PLBART is pre-trained on an extensive collection of Java and Python functions and associated NL text via denoising autoencoding. Experiments on code summarization in the English language, code generation, and code translation in seven programming languages show that PLBART outperforms or rivals state-of-the-art models. Moreover, experiments on discriminative tasks, e.g., program repair, clone detection, and vulnerable code detection, demonstrate PLBART's effectiveness in program understanding. Furthermore, analysis reveals that PLBART learns program syntax, style (e.g., identifier naming convention), logical flow (e.g., if block inside an else block is equivalent to else if block) that are crucial to program semantics and thus excels even with limited annotations.

Citations

• A Convolutional Attention Network for Extreme Summarization of Source Code
• Tree-Transformer: A Transformer-Based Method for Correction of Tree-Structured Data
• Release Strategies and the Social Impacts of Language Models
• Unsupervised Translation of Programming Languages
• GraphCodeBERT: Pre-training Code Representations with Data Flow
• CodeBLEU: a Method for Automatic Evaluation of Code Synthesis
• CodeBERT: A Pre-Trained Model for Programming and Natural Languages
• CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding\n and Generation

Cited by

• Rethinking the Capability of Fine-Tuned Language Models for Automated Vulnerability Repair
• Enhancing Code Understanding for Impact Analysis by Combining Transformers and Program Dependence Graphs
• Evaluating and Mitigating the Misguidance Effect of Buggy Code in LLM-Generated Unit Tests
• Beyond Language Boundaries: Uncovering Programming Language Families for Code Language Models
• SoK: Understanding (New) Security Issues Across AI4Code Use Cases
• Beyond Code Pairs: Dialogue-Based Data Generation for LLM Code Translation
• Agentic Program Verification
• Towards a Human-in-the-Loop Framework for Reliable Patch Evaluation Using an LLM-as-a-Judge
• A Metamorphic Testing Perspective on Knowledge Distillation for Language Models of Code: Does the Student Deeply Mimic the Teacher?
• Enhancing Neural Code Representation with Additional Context
• Learning Bug Context for PyTorch-to-JAX Translation with LLMs
• Multi Language Models for On-the-Fly Syntax Highlighting
• SecureBERT 2.0: Advanced Language Model for Cybersecurity Intelligence
• Red Teaming Program Repair Agents: When Correct Patches can Hide Vulnerabilities
• What Do They Fix? LLM-Aided Categorization of Security Patches for Critical Memory Bugs
• Literature review on vulnerability detection using NLP technology
• Automated Repair of C Programs Using Large Language Models
• Universal and Transferable Adversarial Attack on Large Language Models Using Exponentiated Gradient Descent
• Model Compression vs. Adversarial Robustness: An Empirical Study on Language Models for Code
• Key-Augmented Neural Triggers for Knowledge Sharing
• Interpreting Performance Profiles with Deep Learning
• SPENCER: Self-Adaptive Model Distillation for Efficient Code Retrieval
• A Systematic Literature Review on Detecting Software Vulnerabilities with Large Language Models
• On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey
• Fine-Tuning Multilingual Language Models for Code Review: An Empirical Study on Industrial C# Projects
• AccessGuru: Leveraging LLMs to Detect and Correct Web Accessibility Violations in HTML Code

Related

• Summarize and Generate to Back-translate: Unsupervised Translation of Programming Languages
• Exploring Data Augmentation for Code Generation Tasks
• NatGen: Generative pre-training by "Naturalizing" source code
• CoTexT: Multi-task Learning with Code-Text Transformer
• AVATAR: A Parallel Corpus for Java-Python Program Translation