2022/06/02 by Toufique Ahmed, Ahmed, Toufique, Prémkumar Dévanbu +2 · 1 citation
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Software Engineering (cs.SE) #Software Engineering Research #Software Reliability and Analysis Research #Software System Performance and Reliability #cs.LG #cs.SE
paper · pdf · doi:10.48550/arxiv.2206.00804
arxiv created 2022/06/02 · openalex publication_date 2022/06/02 · arxiv updated 2022/06/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Foundation models (e.g., CodeBERT, GraphCodeBERT, CodeT5) work well for many software engineering tasks. These models are pre-trained (using self-supervision) with billions of code tokens, and then fine-tuned with hundreds of thousands of labeled examples, typically drawn from many projects. However, software phenomena can be very project-specific. Vocabulary, and other phenomena vary substantially with each project. Thus, training on project-specific data, and testing on the same project, is a promising idea. This hypothesis has to be evaluated carefully, e.g., in a time-series setting, to prevent training-test leakage. We compare several models and training approaches, including same-project training, cross-project training, training a model especially designed to be sample efficient (and thus prima facie well-suited for learning in a limited-sample same-project setting) and a maximalist hybrid approach, fine-tuning first on many projects in many languages and then training on the same-project. We find that the maximalist hybrid setting provides consistent, substantial gains over the state-of-the-art, on many different projects in both Java and Python.