CodeBERT: A Pre-Trained Model for Programming and Natural Languages

2020/01/01 by Zhangyin Feng, Daya Guo, Duyu Tang +8 · 157 citations
Computer Science · #Natural Language Processing Techniques #Topic Modeling #Web Data Mining and Analysis

paper · pdf · doi:10.18653/v1/2020.findings-emnlp.139

openalex publication_date 2020/01/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/31

Abstract

Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, Ming Zhou. Findings of the Association for Computational Linguistics: EMNLP 2020. 2020.

Citations

• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Exploiting Generative AI to Scale up Intelligent Tutoring Systems
• CodeSearchNet Challenge: Evaluating the State of Semantic Code Search

Cited by

• Investigating The Smells of LLM Generated Code
• Large Language Models for Software Engineering: A Systematic Literature Review
• Evolving to Find Optimizations Humans Miss: Using Evolutionary Computation to Improve GPU Code for Bioinformatics Applications
• Supporting Workflow Reproducibility by Linking Bioinformatics Tools across Papers and Executable Code
• MRCoder: An Efficient Context Selecting Approach for Repository-Level Code Generation
• Large Language Model for Vulnerability Detection and Repair: Literature Review and the Road Ahead
• AI for DevSecOps: A Landscape and Future Opportunities
• Dissecting the Black Box: Circuit-Level Analysis of LLM Vulnerability Detection
• MANDO-LLM: Heterogeneous Graph Transformers with Large Language Models for Smart Contract Vulnerability Detection
• An Empirical Study of the Non-Determinism of ChatGPT in Code Generation
• Program Synthesis with Large Language Models
• Readability-Robust Code Summarization via Meta Curriculum Learning
• Optimizing Knowledge Utilization for Multi-Intent Comment Generation with Large Language Models
• CodeWiki: Evaluating AI's Ability to Generate Holistic Documentation for Large-Scale Codebases
• MAGNET: A Multi-Graph Attentional Network for Code Clone Detection
• MLCPD: A Unified Multi-Language Code Parsing Dataset with Universal AST Schema
• Understanding Self-Admitted Technical Debt in Test Code: An Empirical Study
• Wisdom and Delusion of LLM Ensembles for Code Generation and Repair
• R2ComSync: Improving Code-Comment Synchronization with In-Context Learning and Reranking
• Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Models
• Practical Code RAG at Scale: Task-Aware Retrieval Design Choices under Compute Budgets
• Unified Pre-training for Program Understanding and Generation
• Selecting and Combining Large Language Models for Scalable Code Clone Detection
• SBAN: A Framework & Multi-Dimensional Dataset for Large Language Model Pre-Training and Software Code Mining
• DuoLens: A Framework for Robust Detection of Machine-Generated Multilingual Text and Code
• SpecAgent: A Speculative Retrieval and Forecasting Agent for Code Completion
• HGAdapter: Hypergraph-based Adapters in Language Models for Code Summarization and Clone Detection
• Reasoning Distillation and Structural Alignment for Improved Code Generation
• All You Need is One: Capsule Prompt Tuning with a Single Vector
• Rewiring Experts on the Fly:Continuous Rerouting for Better Online Adaptation in Mixture-of-Expert models
• Beyond Function-Level Search: Repository-Aware Dual-Encoder Code Retrieval with Adversarial Verification
• A Systematic Study of Time Limit Exceeded Errors in Online Programming Assignments
• Signature in Code Backdoor Detection, how far are we?
• A Hierarchical Quantized Tokenization Framework for Task-Adaptive Graph Representation Learning
• Enhancing Neural Code Representation with Additional Context
• Evaluating Line-level Localization Ability of Learning-based Code Vulnerability Detection Models
• RepoSummary: Feature-Oriented Summarization and Documentation Generation for Code Repositories
• Scalable and Explainable Enterprise Knowledge Discovery Using Graph-Centric Hybrid Retrieval
• CLASP: Training-Free LLM-Assisted Source Code Watermarking via Semantic-Preserving Transformations
• ECO: Enhanced Code Optimization via Performance-Aware Prompting for Code-LLMs
• The Hidden DNA of LLM-Generated JavaScript: Structural Patterns Enable High-Accuracy Authorship Attribution
• Towards Speeding up Program Repair with Non-Autoregressive Model
• TIT: A Tree-Structured Instruction Tuning Approach for LLM-Based Code Translation
• LLM Based Long Code Translation using Identifier Replacement
• RAG4Tickets: AI-Powered Ticket Resolution via Retrieval-Augmented Generation on JIRA and GitHub Data
• Fortifying LLM-Based Code Generation with Graph-Based Reasoning on Secure Coding Practices
• Beyond More Context: How Granularity and Order Drive Code Completion Quality
• DeepV: A Model-Agnostic Retrieval-Augmented Framework for Verilog Code Generation with a High-Quality Knowledge Base
• Retrieval-Augmented Code Generation: A Survey with Focus on Repository-Level Approaches
• NatGVD: Natural Adversarial Example Attack towards Graph-based Vulnerability Detection
• Detecting Semantic Clones of Unseen Functionality
• Learning Representations Through Contrastive Neural Model Checking
• MulVuln: Enhancing Pre-trained LMs with Shared and Language-Specific Knowledge for Multilingual Vulnerability Detection
• Multi Language Models for On-the-Fly Syntax Highlighting
• GramTrans: A Better Code Representation Approach in Code Generation
• C2|Q>: A Robust Framework for Bridging Classical and Quantum Software Development
• Model-Agnostic Correctness Assessment for LLM-Generated Code via Dynamic Internal Representation Selection
• Analyzing Latent Concepts in Code Language Models
• LongCodeZip: Compress Long Context for Code Language Models
• Evaluating Large Language Models Trained on Code
• Which Programming Language and Model Work Best With LLM-as-a-Judge For Code Retrieval?
• MAVUL: Multi-Agent Vulnerability Detection via Contextual Reasoning and Interactive Refinement
• SecureBERT 2.0: Advanced Language Model for Cybersecurity Intelligence
• Are Classical Clone Detectors Good Enough For the AI Era?
• DeepCodeSeek: Real-Time API Retrieval for Context-Aware Code Generation
• JSProtect: A Scalable Obfuscation Framework for Mini-Games in WeChat
• Eliciting and Improving the Causal Reasoning Abilities of Large Language Models with Conditional Statements
• A Survey on Backdoor Threats in Large Language Models (LLMs): Attacks, Defenses, and Evaluation Methods
• TusoAI: Agentic Optimization for Scientific Methods
• RANGER -- Repository-Level Agent for Graph-Enhanced Retrieval
• VeriGRAG: Enhancing LLM-Based Verilog Code Generation with Structure-Aware Soft Prompts
• Towards Human-interpretable Explanation in Code Clone Detection using LLM-based Post Hoc Explainer
• What Do They Fix? LLM-Aided Categorization of Security Patches for Critical Memory Bugs
• PseudoBridge: Pseudo Code as the Bridge for Better Semantic and Logic Alignment in Code Retrieval
• Fine-Tuning LLMs to Analyze Multiple Dimensions of Code Review: A Maximum Entropy Regulated Long Chain-of-Thought Approach
• Synergistic Enhancement of Requirement-to-Code Traceability: A Framework Combining Large Language Model based Data Augmentation and an Advanced Encoder
• Enhancing LLM-based Fault Localization with a Functionality-Aware Retrieval-Augmented Generation Framework
• SCOPE: Synthetic Conditional Objectives for Policy Evolution in Black-Box Combinatorial Optimization
• Security and Privacy Challenges of Large Language Models: A Survey
• Previously on... Automating Code Review
• LLM-based Vulnerability Discovery through the Lens of Code Metrics
• CCQA: Generating Question from Solution Can Improve Inference-Time Reasoning in SLMs
• LLaVul: A Multimodal LLM for Interpretable Vulnerability Reasoning about Source Code
• SLICET5: Static Program Slicing using Language Models with Copy Mechanism and Constrained Decoding
• Revisiting Vulnerability Patch Localization: An Empirical Study and LLM-Based Solution
• Brevity is the Soul of Wit: Condensing Code Changes to Improve Commit Message Generation
• An Evaluation-Centric Paradigm for Scientific Visualization Agents
• SecureFixAgent: A Hybrid LLM Agent for Automated Python Static Vulnerability Repair
• SWE-QA: Can Language Models Answer Repository-level Code Questions?
• Detecting Struggling Student Programmers using Proficiency Taxonomies
• DSPC: Dual-Stage Progressive Compression Framework for Efficient Long-Context Reasoning
• Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues
• An Empirical Study of Retrieval-Augmented Code Generation: Challenges and Opportunities
• EfficientUICoder: Efficient MLLM-based UI Code Generation via Input and Output Token Compression
• VisDocSketcher: Towards Scalable Visual Documentation with Agentic Systems
• VulAgent: Hypothesis-Validation based Multi-Agent Vulnerability Detection
• Weakly Supervised Vulnerability Localization via Multiple Instance Learning
• Large Language Models for Security Operations Centers: A Comprehensive Survey
• I Know Who Clones Your Code: Interpretable Smart Contract Similarity Detection
• Cross-Domain Evaluation of Transformer-Based Vulnerability Detection on Open & Industry Data
• Probing Pre-trained Language Models on Code Changes: Insights from ReDef, a High-Confidence Just-in-Time Defect Prediction Dataset
• TriagerX: Dual Transformers for Bug Triaging Tasks with Content and Interaction Based Rankings
• CoSQA: 20,000+ Web Queries for Code Search and Question Answering
• EyeMulator: Improving Code Language Models by Mimicking Human Visual Attention
• GitHub Copilot AI pair programmer: Asset or Liability?
• Fuzz4All: Universal Fuzzing with Large Language Models
• PatchSeeker: Mapping NVD Records to their Vulnerability-fixing Commits with LLM Generated Commits and Embeddings
• How Small Transformation Expose the Weakness of Semantic Similarity Measures
• When Code Crosses Borders: A Security-Centric Study of LLM-based Code Translation
• Code Review Without Borders: Evaluating Synthetic vs. Real Data for Review Recommendation
• Automated Repair of C Programs Using Large Language Models
• Beyond the Surface: A Solution-Aware Retrieval Model for Competition-level Code Generation
• VULSOLVER: Vulnerability Detection via LLM-Driven Constraint Solving
• CoTexT: Multi-task Learning with Code-Text Transformer
• Detecting Stealthy Data Poisoning Attacks in AI Code Generators
• Efficient Code Embeddings from Code Generation Models
• Deep Learning Based Concurrency Bug Detection and Localization
• BridgeShield: Enhancing Security for Cross-chain Bridge Applications via Heterogeneous Graph Mining
• Smart Contract Intent Detection with Pre-trained Programming Language Model
• Functional Consistency of LLM Code Embeddings: A Self-Evolving Data Synthesis Framework for Benchmarking
• SIExVulTS: Sensitive Information Exposure Vulnerability Detection System using Transformer Models and Static Analysis
• Stack Trace-Based Crash Deduplication with Transformer Adaptation
• VISION: Robust and Interpretable Code Vulnerability Detection Leveraging Counterfactual Augmentation
• FALCON: Autonomous Cyber Threat Intelligence Mining with LLMs for IDS Rule Generation
• An Empirical Study of Knowledge Distillation for Code Understanding Tasks
• LinkAnchor: An Autonomous LLM-Based Agent for Issue-to-Commit Link Recovery
• “It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers
• NeMo: A Neuron-Level Modularizing-While-Training Approach for Decomposing DNN Models
• Hallucination in LLM-Based Code Generation: An Automotive Case Study
• Tabularis Formatus: Predictive Formatting for Tables
• Data and Context Matter: Towards Generalizing AI-based Software Vulnerability Detection
• Dataset Construction for Training LLM to Learn Analog Circuit Knowledge
• Code Vulnerability Detection Across Different Programming Languages with AI Models
• SaraCoder: Orchestrating Semantic and Structural Cues for Resource-Optimized Repository-Level Code Completion
• OmniLLP: Enhancing LLM-based Log Level Prediction with Context-Aware Retrieval
• What Builds Effective In-Context Examples for Code Generation?
• Klear-CodeTest: Scalable Test Case Generation for Code Reinforcement Learning
• Model Compression vs. Adversarial Robustness: An Empirical Study on Language Models for Code
• Analyzing Prominent LLMs: An Empirical Study of Performance and Complexity in Solving LeetCode Problems
• StoneDetector: Conventional and versatile code clone detection for Java
• SmartLLMs Scheduler: A Framework for Cost-Effective LLMs Utilization
• On the Evaluation of Large Language Models in Multilingual Vulnerability Repair
• HyCodePolicy: Hybrid Language Controllers for Multimodal Monitoring and Decision in Embodied Agents
• An Efficient and Adaptive Next Edit Suggestion Framework with Zero Human Instructions in IDEs
• Meta-RAG on Large Codebases Using Code Summarization
• NATLM: Detecting Defects in NFT Smart Contracts Leveraging LLM
• Interpreting Performance Profiles with Deep Learning
• SPENCER: Self-Adaptive Model Distillation for Efficient Code Retrieval
• A Systematic Literature Review on Detecting Software Vulnerabilities with Large Language Models
• RePaCA: Leveraging Reasoning Large Language Models for Static Automated Patch Correctness Assessment
• IFEvalCode: Controlled Code Generation
• Breaking Obfuscation: Cluster-Aware Graph with LLM-Aided Recovery for Malicious JavaScript Detection
• SAEL: Leveraging Large Language Models with Adaptive Mixture-of-Experts for Smart Contract Vulnerability Detection
• Fine-Tuning Code Language Models to Detect Cross-Language Bugs
• Out of Distribution, Out of Luck: How Well Can LLMs Trained on Vulnerability Datasets Detect Top 25 CWE Weaknesses?
• MultiAIGCD: A Comprehensive dataset for AI Generated Code Detection Covering Multiple Languages, Models,Prompts, and Scenarios
• On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey

Related

• CodeBERT: A Pre-Trained Model for Programming and Natural Languages
• CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code
• CodeBERT-nt: code naturalness via CodeBERT
• Which Features are Learned by CodeBert: An Empirical Study of the BERT-based Source Code Representation Learning
• CodeGemma: Open Code Models Based on Gemma