From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debugging

2024/10/02 by Yuling Shi, Shi, Yuling, S. ShouHan Wang +5 · 17 citations
Computer Science · #Advanced Software Engineering Methodologies #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Model-Driven Software Engineering Techniques #Programming Languages (cs.PL) #Software Engineering (cs.SE) #Software Engineering Research

paper · pdf · doi:10.48550/arxiv.2410.01215

openalex publication_date 2024/10/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

While large language models have made significant strides in code generation, the pass rate of the generated code is bottlenecked on subtle errors, often requiring human intervention to pass tests, especially for complex problems. Existing LLM-based debugging systems treat generated programs as monolithic units, failing to address bugs at multiple levels of granularity, from low-level syntax errors to high-level algorithmic flaws. In this paper, we introduce Multi-Granularity Debugger (MGDebugger), a hierarchical code debugger by isolating, identifying, and resolving bugs at various levels of granularity. MGDebugger decomposes problematic code into a hierarchical tree structure of subfunctions, with each level representing a particular granularity of error. During debugging, it analyzes each subfunction and iteratively resolves bugs in a bottom-up manner. To effectively test each subfunction, we propose an LLM-simulated Python executor, which traces code execution and tracks important variable states to pinpoint errors accurately. Extensive experiments demonstrate that MGDebugger outperforms existing debugging systems, achieving an 18.9% improvement in accuracy over seed generations in HumanEval and a 97.6% repair success rate in HumanEvalFix. Furthermore, MGDebugger effectively fixes bugs across different categories and difficulty levels, demonstrating its robustness and effectiveness.

Cited by

• Analyzing Code Injection Attacks on LLM-based Multi-Agent Systems in Software Development
• Fed-SE: Federated Self-Evolution for Privacy-Constrained Multi-Environment LLM Agents
• SLMFix: Leveraging Small Language Models for Error Fixing with Reinforcement Learning
• The Architecture of Errors: From Universal Impossibility to Patch-Local LLM Reliability
• Readability-Robust Code Summarization via Meta Curriculum Learning
• HyperAgent: Leveraging Hypergraphs for Topology Optimization in Multi-Agent Communication
• D3MAS: Decompose, Deduce, and Distribute for Enhanced Knowledge Sharing in Multi-Agent Systems
• GraphTracer: Graph-Guided Failure Tracing in LLM Agents for Robust Multi-Turn Deep Search
• LongCodeZip: Compress Long Context for Code Language Models
• Multi-LLM Orchestration for High-Quality Code Generation: Exploiting Complementary Model Strengths
• SWE-QA: Can Language Models Answer Repository-level Code Questions?
• RepoDebug: Repository-Level Multi-Task and Multi-Language Debugging Evaluation of Large Language Models
• ReCode: Improving LLM-based Code Repair with Fine-Grained Retrieval-Augmented Generation
• Pruning the Unsurprising: Efficient LLM Reasoning via First-Token Surprisal
• EvoC2Rust: A Skeleton-guided Framework for Project-Level C-to-Rust Translation
• SWE-Exp: Experience-Driven Software Issue Resolution
• SWE-Debate: Competitive Multi-Agent Debate for Software Issue Resolution

Related

• Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-step
• DebugBench: Evaluating Debugging Capability of Large Language Models
• Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?
• MdEval: Massively Multilingual Code Debugging
• InspectCoder: Dynamic Analysis-Enabled Self Repair through interactive LLM-Debugger Collaboration