Assessing LLM code generation quality through path planning tasks

2025/04/30 by Wanyi Chen, Chen, Wanyi, Meng-Wen Su +3
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Robotic Path Planning Algorithms #Safety Systems Engineering in Autonomy #Software Engineering (cs.SE) #Software Testing and Debugging Techniques

paper · pdf · doi:10.48550/arxiv.2504.21276

openalex publication_date 2025/04/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

As LLM-generated code grows in popularity, more evaluation is needed to assess the risks of using such tools, especially for safety-critical applications such as path planning. Existing coding benchmarks are insufficient as they do not reflect the context and complexity of safety-critical applications. To this end, we assessed six LLMs' abilities to generate the code for three different path-planning algorithms and tested them on three maps of various difficulties. Our results suggest that LLM-generated code presents serious hazards for path planning applications and should not be applied in safety-critical contexts without rigorous testing.

Citations

• SWE-Bench+: Enhanced Coding Benchmark for LLMs
• A Survey on Large Language Models for Code Generation
• Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code
• Bugs in Large Language Models Generated Code: An Empirical Study
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• Model evaluation for extreme risks
• A Hazard Analysis Framework for Code Synthesis Large Language Models
• Evaluating Large Language Models Trained on Code
• An Algorithm for Finding Best Matches in Logarithmic Expected Time
• A Note on Two Problems in Connexion with Graphs

Related

• Quality Assurance of LLM-generated Code: Addressing Non-Functional Quality Characteristics
• Automatic Generation of Benchmarks and Reliable LLM Judgment for Code Tasks
• What's Wrong with Your Code Generated by Large Language Models? An Extensive Study
• Is LLM-Generated Code More Maintainable & Reliable than Human-Written Code?
• OSS-Bench: Benchmark Generator for Coding LLMs