2018/12/16 by Miltiadis Allamanis, Allamanis, Miltiadis · 4 citations
Computer Science · #Advanced Malware Detection Techniques #FOS: Computer and information sciences #Machine Learning (cs.LG) #Software Engineering (cs.SE) #Software Engineering Research #Software Reliability and Analysis Research
paper · pdf · doi:10.48550/arxiv.1812.06469
openalex publication_date 2018/12/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
The field of big code relies on mining large corpora of code to perform some\nlearning task. A significant threat to this approach has been recently\nidentified by Lopes et al. (2017) who found a large amount of near-duplicate\ncode on GitHub. However, the impact of code duplication has not been noticed by\nresearchers devising machine learning models for source code. In this work, we\nexplore the effects of code duplication on machine learning models showing that\nreported performance metrics are sometimes inflated by up to 100% when testing\non duplicated code corpora compared to the performance on de-duplicated corpora\nwhich more accurately represent how machine learning models of code are used by\nsoftware engineers. We present a duplication index for widely used datasets,\nlist best practices for collecting code corpora and evaluating machine learning\nmodels on them. Finally, we release tools to help the community avoid this\nproblem in future research.\n