Performance Review on LLM for solving leetcode problems

2025/02/16 by Wang, Lun, Shi, Chuanqi, Du, Shaoshui +5 · 2 citations
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Software Engineering (cs.SE)

paper · doi:10.48550/arxiv.2502.15770

Abstract

This paper presents a comprehensive performance evaluation of Large Language Models (LLMs) in solving programming challenges from Leetcode, a widely used platform for algorithm practice and technical interviews. We began by crawling the Leetcode website to collect a diverse set of problems encompassing various difficulty levels and topics. Using this dataset, we generated solutions with multiple LLMs, including GPT-4 and GPT-3.5-turbo (ChatGPT-turbo). The generated solutions were systematically evaluated for correctness and efficiency. We employed the pass@k metric to assess the success rates within a given number of attempts and analyzed the runtime performance of the solutions. Our results highlight the strengths and limitations of current LLMs [10] in code generation and problem-solving tasks, providing insights into their potential applications and areas for improvement in automated programming assistance.

Cited by

• Understanding the Role of Large Language Models in Competitive Programming
• Reward-Shifted Speculative Sampling Is An Efficient Test-Time Weak-to-Strong Aligner

Related

• Analyzing Prominent LLMs: An Empirical Study of Performance and Complexity in Solving LeetCode Problems
• Holistic Evaluation of State-of-the-Art LLMs for Code Generation
• On Evaluating the Efficiency of Source Code Generated by LLMs
• Evaluating the Limitations of Local LLMs in Solving Complex Programming Challenges
• LLM-ProS: Analyzing Large Language Models' Performance in Competitive Problem Solving