LongCoder: A Long-Range Pre-trained Language Model for Code Completion

2023/06/26 by Daya Guo, Canwen Xu, Guo, Daya +7 · 24 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computational Physics and Python Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Software Engineering (cs.SE) #Software Engineering Research #Software System Performance and Reliability

paper · pdf · doi:10.48550/arxiv.2306.14893

openalex publication_date 2023/06/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we introduce a new task for code completion that focuses on handling long code input and propose a sparse Transformer model, called LongCoder, to address this task. LongCoder employs a sliding window mechanism for self-attention and introduces two types of globally accessible tokens - bridge tokens and memory tokens - to improve performance and efficiency. Bridge tokens are inserted throughout the input sequence to aggregate local information and facilitate global interaction, while memory tokens are included to highlight important statements that may be invoked later and need to be memorized, such as package imports and definitions of classes, functions, or structures. We conduct experiments on a newly constructed dataset that contains longer code context and the publicly available CodeXGLUE benchmark. Experimental results demonstrate that LongCoder achieves superior performance on code completion tasks compared to previous models while maintaining comparable efficiency in terms of computational resources during inference. All the codes and data are available at https://github.com/microsoft/CodeBERT.

Cited by

• AgentWatcher: A Rule-based Prompt Injection Monitor
• Accelerate Speculative Decoding with Sparse Computation in Verification
• SoK: Understanding (New) Security Issues Across AI4Code Use Cases
• Framework-Aware Code Generation with API Knowledge Graph-Constructed Data: A Study on HarmonyOS
• A Viable Paradigm of Software Automation: Iterative End-to-End Automated Software Development
• Understanding and Improving Length Generalization in Hierarchical Sparse Attention Models
• QuanBench: Benchmarking Quantum Code Generation with Large Language Models
• On Pretraining for Project-Level Code Completion
• Taming the Fragility of KV Cache Eviction in LLM Inference
• A Matter of Representation: Towards Graph-Based Abstract Code Generation
• LongCodeZip: Compress Long Context for Code Language Models
• SemShareKV: Efficient KVCache Sharing for Semantically Similar Prompts via Token-Level LSH Matching
• SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving
• Intuition to Evidence: Measuring AI's True Impact on Developer Productivity
• Mamba Modulation: On the Length Generalization of Mamba
• SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfer
• Value-Guided KV Compression for LLMs via Approximated CUR Decomposition
• LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocation
• EvolKV: Evolutionary KV Cache Compression for LLM Inference
• SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruning
• AdaptCache: KV Cache Native Storage Hierarchy for Low-Delay and High-Quality Language Model Serving
• Your Coding Intent is Secretly in the Context and You Should Deliberately Infer It Before Completion
• Impact-driven Context Filtering For Cross-file Code Completion
• Smooth Reading: Bridging the Gap of Recurrent LLM to Self-Attention LLM on Long-Context Tasks

Related

• LongCodeZip: Compress Long Context for Code Language Models
• SparseCoder: Advancing Source Code Analysis with Sparse Attention and Learned Token Pruning
• Bridge and Hint: Extending Pre-trained Language Models for Long-Range Code
• Long Code Arena: a Set of Benchmarks for Long-Context Code Models
• LEANCODE: Understanding Models Better for Code Simplification of Pre-trained Large Language Models