2018/12/16 by Miltiadis Allamanis, Allamanis, Miltiadis · 13 citations
Computer Science · #Advanced Malware Detection Techniques #FOS: Computer and information sciences #Machine Learning (cs.LG) #Software Engineering (cs.SE) #Software Engineering Research #Software Reliability and Analysis Research #cs.LG #cs.SE
paper · pdf · doi:10.48550/arxiv.1812.06469
Published in SPLASH Onward! 2019
openalex publication_date 2018/12/16 · arxiv created 2019/08/11 · arxiv updated 2019/08/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
The field of big code relies on mining large corpora of code to perform some learning task. A significant threat to this approach has been recently identified by Lopes et al. (2017) who found a large amount of near-duplicate code on GitHub. However, the impact of code duplication has not been noticed by researchers devising machine learning models for source code. In this work, we explore the effects of code duplication on machine learning models showing that reported performance metrics are sometimes inflated by up to 100% when testing on duplicated code corpora compared to the performance on de-duplicated corpora which more accurately represent how machine learning models of code are used by software engineers. We present a duplication index for widely used datasets, list best practices for collecting code corpora and evaluating machine learning models on them. Finally, we release tools to help the community avoid this problem in future research.