2022/03/16 by Zhiruo Wang, Grace Cuenca, Wang, Zhiruo +7 · 5 citations
Computer Science · #Advanced Malware Detection Techniques #Computation and Language (cs.CL) #FOS: Computer and information sciences #Software Engineering Research #Software Testing and Debugging Techniques
paper · pdf · doi:10.48550/arxiv.2203.08388
openalex publication_date 2022/03/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
While there has been a recent burgeoning of applications at the intersection of natural and programming languages, such as code generation and code summarization, these applications are usually English-centric. This creates a barrier for program developers who are not proficient in English. To mitigate this gap in technology development across languages, we propose a multilingual dataset, MCoNaLa, to benchmark code generation from natural language commands extending beyond English. Modeled off of the methodology from the English Code/Natural Language Challenge (CoNaLa) dataset, we annotated a total of 896 NL-code pairs in three languages: Spanish, Japanese, and Russian. We present a quantitative evaluation of performance on the MCoNaLa dataset by testing with state-of-the-art code generation systems. While the difficulties vary across these three languages, all systems lag significantly behind their English counterparts, revealing the challenges in adapting code generation to new languages.