2024/08/31 by Adrian Marius Dumitran, Dumitran, Adrian Marius, Adrian Cǎtǎlin Badea +3
Social Sciences · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Higher Education Learning Practices #Programming Languages (cs.PL) #Software Engineering (cs.SE)
paper · pdf · doi:10.48550/arxiv.2409.09054
openalex publication_date 2024/08/31 · openalex created_date 2024/10/23 · openalex updated_date 2026/07/28
This study explores the performance of large language models (LLMs) in solving competitive programming problems from the Romanian Informatics Olympiad at the county level. Romania, a leading nation in computer science competitions, provides an ideal environment for evaluating LLM capabilities due to its rich history and stringent competition standards. We collected and analyzed a dataset comprising 304 challenges from 2002 to 2023, focusing on solutions written by LLMs in C++ and Python for these problems. Our primary goal is to understand why LLMs perform well or poorly on different tasks. We evaluated various models, including closed-source models like GPT-4 and open-weight models such as CodeLlama and RoMistral, using a standardized process involving multiple attempts and feedback rounds. The analysis revealed significant variations in LLM performance across different grades and problem types. Notably, GPT-4 showed strong performance, indicating its potential use as an educational tool for middle school students. We also observed differences in code quality and style across various LLMs