2024/02/20 by Jiayi Lin, Hande Dong, Lin, Jiayi +5 · 1 citation
Computer Science · #Advanced Computational Techniques and Applications #FOS: Computer and information sciences #Software Engineering (cs.SE)
paper · pdf · doi:10.48550/arxiv.2402.12813
openalex publication_date 2024/02/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
The scaling law is becoming a fundamental law in many machine learning areas. That is, test error falls off with the power law when increasing training data, model size, and computing resource. However, whether this law is suitable for the task of code understanding is not well studied, and most current language models for code understanding are about 100M parameters, which are relatively "small" compared to large language models. In this paper, we conduct extensive experiments to investigate the scaling law for the code understanding task by varying training data, model size, and computing resource. We validate that the test error of code understanding models falls off with the power law when using larger models, indicating that the scaling law is suitable for the code understanding task. Besides, we apply different scales of models to two downstream code understanding tasks, and find that the performance increases with larger scale of models. Finally, we train a large-scale code understanding model named CoLSBERT with 1.5B parameters on a large dataset using more computing resource, which outperforms previous work by a large margin. We will release our code and the CoLSBERT model when our paper is published.