Teaching Large Language Models to Self-Debug
2023/04/11 by Xinyun Chen, Maxwell Lin, Chen, Xinyun +5 · 2 voices · 245 citations
Computer Science · #Algorithmic program debugging #Benchmark (surveying) #Compiler #Computer science #Correctness #Debugging #Parallel Computing and Optimization Techniques #Programming language #Python (programming language) #Software #Software Engineering Research #Topic Modeling #Unit testing
paper · pdf · doi:10.48550/arxiv.2304.05128
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2023/04/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/04
Abstract
Large language models (LLMs) have achieved impressive performance on code generation. However, for complex programming tasks, generating the correct solution in one go becomes challenging, thus some prior works have designed program repair approaches to improve code generation performance. In this work, we propose Self-Debugging, which teaches a large language model to debug its predicted program via few-shot demonstrations. In particular, we demonstrate that Self-Debugging can teach the large language model to perform rubber duck debugging; i.e., without any human feedback on the code correctness or error messages, the model is able to identify its mistakes by investigating the execution results and explaining the generated code in natural language. Self-Debugging achieves the state-of-the-art performance on several code generation benchmarks, including the Spider dataset for text-to-SQL generation, TransCoder for C++-to-Python translation, and MBPP for text-to-Python generation. On the Spider benchmark where there are no unit tests to verify the correctness of predictions, Self-Debugging with code explanation consistently improves the baseline by 2-3%, and improves the prediction accuracy on problems of the hardest level by 9%. On TransCoder and MBPP where unit tests are available, Self-Debugging improves the baseline accuracy by up to 12%. Meanwhile, by leveraging feedback messages and reusing failed predictions, Self-Debugging notably improves sample efficiency, and can match or outperform baseline models that generate more than 10x candidate programs.
Cited by
- Beyond Text-to-SQL: Can LLMs Really Debug Enterprise ETL SQL?
- Structured Feedback Improves Repair in an LLM Agent Loop
- Cross-Model LLM Code Review: Should you use Claude to review Codex or vice versa?
- Bridging the Last Mile of Circuit Design: PostEDA-Bench, a Hierarchical Benchmark for PPA Convergence and DRC Fixing
- CodeRescue: Budget-Calibrated Recovery Routing for Coding Agents
- DataFlow-Harness: A Grounded Code-Agent Platform for Constructing Editable LLM Data Pipelines
- Fully Automated End-to-End Adversary Emulation from MITRE ATT&CK Based Cyber Threat Intelligence Using LLMs
- CIPHER: A Decoupled Exploration-Selection Framework for Test-Time Scaling of Data Science Agents
- Models Can Model, But Can't Bind: Structured Grounding in Text-to-Optimization
- ExecuGraph: A Multi-Agent, Execution-Grounded Framework for Reliable Backend Code Synthesis with Large Language Models
- RAIL Guard: Closing the Evaluation-to-Remediation Gap in Responsible AI for LLM Agents
- VeriSimpl: Robust Optimization Modeling from Natural Language using Simplification-based Verification
- FormulaSPIN: Self-Play Fine-Tuning for Natural Language to Spreadsheet Formula Generation
- LLM-Based Agentic Systems for Software Engineering: Challenges and Opportunities
- Reasoning Models Reason Well, Until They Don't
- EditLord: Learning Code Transformation Rules for Code Editing
- Gold-medalist Performance in Solving Olympiad Geometry with AlphaGeometry2
- Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges
- NERFIFY: A Multi-Agent Framework for Turning NeRF Papers into Code
- DualityCert: Verifier-Gated Language-Model Repair of Broken Duality Claims in Quantum Field Theory
- Multi-level Code Optimization via Mixture of Prompts
- Specula: Scaling formal specifications for autonomous model checking of system code
- Beyond "What to Retrieve": Uncertainty in Retrieval-Augmented Code Generation
- Procedural Knowledge at Scale Improves Reasoning
- AutoBaxBuilder: Bootstrapping Code Security Benchmarking
- VIGOR+: Iterative Confounder Generation and Validation via LLM-CEVAE Feedback Loop
- SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios
- SoK: Understanding (New) Security Issues Across AI4Code Use Cases
- A Network Arena for Benchmarking AI Agents on Network Troubleshooting
- FutureWeaver: Planning Test-Time Compute for Multi-Agent Systems with Modularized Collaboration
- CogMCTS: A Novel Cognitive-Guided Monte Carlo Tree Search Framework for Iterative Heuristic Evolution with Large Language Models
- VIGIL: A Reflective Runtime for Self-Healing Agents
- Mitigating Self-Preference by Authorship Obfuscation
- RefineBench: Evaluating Refinement Capability of Language Models via Checklists
- On the Limits of Test-Time Compute: Sequential Reward Filtering for Better Inference
- SAGE: Semantic-Aware Gray-Box Game Regression Testing with Large Language Models
- Generating Verifiable Chain of Thoughts from Exection-Traces
- EWE: An Agentic Framework for Extreme Weather Analysis
- Progress by Pieces: Test-Time Scaling for Autoregressive Image Generation
- LLMs as Firmware Experts: A Runtime-Grown Tree-of-Agents Framework
- ELPO: Ensemble Learning Based Prompt Optimization for Large Language Models
- SDA: Steering-Driven Distribution Alignment for Open LLMs without Fine-Tuning
- InfCode: Adversarial Iterative Refinement of Tests and Patches for Reliable Software Issue Resolution
- From Solving to Verifying: A Unified Objective for Robust Reasoning in LLMs
- Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?
- Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talk
- RulePilot: An LLM-Powered Agent for Security Rule Generation
- ExPairT-LLM: Exact Learning for LLM Code Selection by Pairwise Queries
- ACT as Human: Multimodal Large Language Model Data Annotation with Critical Thinking
- MACEval: A Multi-Agent Continual Evaluation Network for Large Models
- Self-Correction Distillation for Structured Data Question Answering
- Retriv at BLP-2025 Task 2: Test-Driven Feedback-Guided Framework for Bangla-to-Python Code Generation
- VeriCoT: Neuro-symbolic Chain-of-Thought Validation via Logical Consistency Checks
- EDIT-Bench: Evaluating LLM Abilities to Perform Real-World Instructed Code Edits
- ReAcTree: Hierarchical LLM Agent Trees with Control Flow for Long-Horizon Task Planning
- The Sequential Edge: Inverse-Entropy Voting Beats Parallel Self-Consistency at Matched Compute
- HarnessLLM: Automatic Testing Harness Generation via Reinforcement Learning
- ORANGE: An Online Reflection ANd GEneration framework with Domain Knowledge for Text-to-SQL
- PDE-SHARP: PDE Solver Hybrids through Analysis and Refinement Passes
- SELF-REDRAFT: Eliciting Intrinsic Exploration-Exploitation Balance in Test-Time Scaling for Code Generation
- Generalizing Test-time Compute-optimal Scaling as an Optimizable Graph
- Debate2Create: Robot Co-design via Large Language Model Debates
- From Role Prompt to Infinite Thinking: Exploiting Persona Conditioning for Inference Cost Attacks in LLMs
- Do Code Language Models Use Tests? A Behavioral and Representational Study of Test-Driven Code Generation
- ClinLens: Towards Long-Horizon Coding Agents for Longitudinal Multimodal Clinical Data Science
- TraceCoder: Explainable and Auditable Code Generation with Position-Key Snippet Versioning
- EsoLang-Bench: Evaluating Genuine Reasoning in Large Language Models via Esoteric Programming Languages
- Lang-PINN: From Language to Physics-Informed Neural Networks via a Multi-Agent Framework
- Flows: Building Blocks of Reasoning and Collaborating AI
- TEXT2DB: Integration-Aware Information Extraction with Large Language Model Agents
- Once Upon an Input: Reasoning via Per-Instance Program Synthesis
- Software Engineering Agents for Embodied Controller Generation : A Study in Minigrid Environments
- Finding the Sweet Spot: Trading Quality, Cost, and Speed During Inference-Time LLM Reflection
- Learning from Supervision with Semantic and Episodic Memory: A Reflective Approach to Agent Adaptation
- Knowledge-Guided Multi-Agent Framework for Application-Level Software Code Generation
- Illusions of reflection: open-ended task reveals systematic failures in Large Language Models' reflective reasoning
- Enhancing geodatabases operability: advanced human-computer interaction through RAG and Multi-Agent Systems
- SpecAgent: A Speculative Retrieval and Forecasting Agent for Code Completion
- An Agentic Framework with LLMs for Solving Complex Vehicle Routing Problems
- LSPO: Length-aware Dynamic Sampling for Policy Optimization in LLM Reasoning
- David vs. Goliath: A comparative study of different-sized LLMs for code generation in the domain of automotive scenario generation
- EvoCAD: Evolutionary CAD Code Generation with Vision Language Models
- DebugTA: An LLM-Based Agent for Simplifying Debugging and Teaching in Programming Education
- Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey
- MUSE-Autoskill: Self-Evolving Agents via Skill Creation, Memory, Management, and Evaluation
- Autonomous Agents for Scientific Discovery: Orchestrating Scientists, Language, Code, and Physics
- Fundamentals of Building Autonomous LLM Agents
- Self-Improving LLM Agents at Test-Time
- Traceability and Accountability in Role-Specialized Multi-Agent LLM Pipelines
- XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitation
- RECODE-H: A Benchmark for Research Code Development with Interactive Human Feedback
- AlphaApollo: A System for Deep Agentic Reasoning
- A Survey of LLM-Based Applications in Programming Education: Balancing Automation and Human Oversight
- A Tale of LLMs and Induced Small Proxies: Scalable Small Language Models for Knowledge Mining
- LLM-Based Multi-Agent Blackboard System for Information Discovery in Data Science
- Towards Reliable Generation of Executable Workflows by Foundation Models
- Multilingual Text-to-SQL: Benchmarking the Limits of Language Models with Collaborative Language Agents
- LLM/Agent-as-Data-Analyst: A Survey
- Large-Scale Constraint Generation -- Can LLMs Parse Hundreds of Constraints?
- From Harm to Help: Turning Reasoning In-Context Demos into Assets for Reasoning LMs
- RepLLM: Toward Automatically Reproducing Network Research Results
- RefineSVG: Visual Feedback-Driven Reinforcement Learning for Image-to-SVG Generation
- SCOPE: Synthetic Conditional Objectives for Policy Evolution in Black-Box Combinatorial Optimization
- Scaling LLM-Driven Multi-Agent Systems: Design Principles and Architectural Scalability Analysis
- Code Driven Planning with Domain-Adaptive Critic
- Actions Speak Louder than Prompts: A Large-Scale Study of LLMs for Graph Inference
- Variation in Verification: Understanding Verification Dynamics in Large Language Models
- Building Data-Driven Occupation Taxonomies: A Bottom-Up Multi-Stage Approach via Semantic Clustering and Multi-Agent Collaboration
- Generating High-Quality Datasets for Code Editing via Open-Source Language Models
- DF-LLaVA: Unlocking MLLMs for Synthetic Image Detection via Knowledge Injection and Conflict-Driven Self-Reflection
- An Empirical Study on Failures in Automated Issue Solving
- VerilogMonkey: Exploring Parallel Scaling for Automated Verilog Code Generation with LLMs
- FVDebug: An LLM-Driven Debugging Assistant for Automated Root Cause Analysis of Formal Verification Failures
- LTA-thinker: Latent Thought-Augmented Training Framework for Large Language Models on Complex Reasoning
- CodeCureAgent: Automatic Classification and Repair of Static Analysis Warnings
- Do Code Semantics Help? A Comprehensive Study on Execution Trace-Based Information for Code Large Language Models
- SQLGovernor: An LLM-powered SQL Toolkit for Real World Application
- PaVeRL-SQL: Text-to-SQL via Partial-Match Rewards and Verbal Reinforcement Learning
- A Probabilistic Inference Scaling Theory for LLM Self-Correction
- ReCode: Improving LLM-based Code Repair with Fine-Grained Retrieval-Augmented Generation
- Aligning Requirement for Large Language Model's Code Generation
- Can Multi-turn Self-refined Single Agent LMs with Retrieval Solve Hard Coding Problems?
- Ensemble Debates with Local Large Language Models for AI Alignment
- A Stitch in Time Saves Nine: Proactive Self-Refinement for Language Models
- Hell or High Water: Evaluating Agentic Recovery from External Failures
- LLM driven Text-to-Table Generation through Sub-Tasks Guidance and Iterative Refinement
- Uncertainty-Aware Semantic Decoding for LLM-Based Sequential Recommendation
- GP and LLMs for Program Synthesis: No Clear Winners
- Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-Correction
- MRG-Bench: Evaluating and Exploring the Requirements of Context for Repository-Level Code Generation
- A Rolling Stone Gathers No Moss: Adaptive Policy Optimization for Stable Self-Evaluation in Large Multimodal Models
- HyCodePolicy: Hybrid Language Controllers for Multimodal Monitoring and Decision in Embodied Agents
- Exploring Direct Instruction and Summary-Mediated Prompting in LLM-Assisted Code Modification
- WarriorMath: Enhancing the Mathematical Ability of Large Language Models with a Defect-aware Framework
- Chatting with your ERP: A Recipe
- R4ec: A Reasoning, Reflection, and Refinement Framework for Recommendation Systems
- ParaStudent: Generating and Evaluating Realistic Student Code by Teaching LLMs to Struggle
- GitChameleon 2.0: Evaluating AI Code Generation Against Python Library Version Incompatibilities
- Learning to Reason Across Parallel Samples for LLM Reasoning
- CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks
- Kodezi Chronos: A Debugging-First Language Model for Repository-Scale Code Understanding
- Stabilizing Black-Box Prompt Optimization with Textual Regularization and Signal Aggregation
- Rethinking Verification for LLM Code Generation: From Generation to Testing
- EasyBCI Agent: Towards Universal Neural Data Preprocessing for Brain-Computer Interfaces
- AuditCoder: Responsibility-Preserving Task Graphs for Auditable Code Generation and Bounded Repair
- Replacing thinking with tool usage enables reasoning in small language models
- Validation Evidence in LLM Repair Agents: How Much of What Passes Actually Tests the Bug?
- Chain-of-Models: Cross-Model Auditing for Bias-Robust LLM Judges
- Reflective Unit Test Generation for Precise Type Error Detection with Large Language Models
- Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation
- Lost at the Beginning of Reasoning
- SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applications
- The Debugging Decay Index: Rethinking Debugging Strategies for Code LLMs
- Use Property-Based Testing to Bridge LLM Code Generation and Validation
- Advanced Applications of Generative AI in Actuarial Science: Case Studies Beyond ChatGPT
- AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientists
- Semantic-Aware Parsing for Security Logs
- Code Reasoning for Software Engineering Tasks: A Survey and A Call to Action
- Strategic Scaling of Test-Time Compute: A Bandit Learning Approach
- Feedback Friction: LLMs Struggle to Fully Incorporate External Feedback
- Execution Guided Line-by-Line Code Generation
- Formalizing Learning from Language Feedback with Provable Guarantees
- PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifier
- Direct Behavior Optimization: Unlocking the Potential of Lightweight LLMs
- Deployability-Centric Infrastructure-as-Code Generation: An LLM-based Iterative Framework
- Sample Complexity and Representation Ability of Test-time Scaling Paradigms
- Truly Self-Improving Agents Require Intrinsic Metacognitive Learning
- Please Translate Again: Two Simple Experiments on Whether Human-Like Reasoning Helps Translation
- A Reasoning-Based Approach to Cryptic Crossword Clue Solving
- Towards Effectively Leveraging Execution Traces for Program Repair with Code LLMs
- Exchange of Perspective Prompting Enhances Reasoning in Large Language Models
- Boosting Open-Source LLMs for Program Repair via Reasoning Transfer and LLM-Guided Reinforcement Learning
- SQLens: An End-to-End Framework for Error Detection and Correction in Text-to-SQL
- A Preference-Driven Methodology for High-Quality Solidity Code Generation
- EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solving
- SysLLMatic: Large Language Models are Software System Optimizers
- Pi-SQL: Enhancing Text-to-SQL with Fine-Grained Guidance from Pivot Programming Languages
- SHARE: An SLM-based Hierarchical Action CorREction Assistant for Text-to-SQL
- AlphaOne: Reasoning Models Thinking Slow and Fast at Test Time
- Knowledge Augmented Complex Problem Solving with Large Language Models: A Survey
- Self-Correcting Code Generation Using Small Language Models
- CADReview: Automatically Reviewing CAD Programs with Error Detection and Correction
- Exploring the Landscape of Text-to-SQL with Large Language Models: Progresses, Challenges and Opportunities
- VulBinLLM: LLM-powered Vulnerability Detection for Stripped Binaries
- UI-Evol: Automatic Knowledge Evolving for Computer Use Agents
- BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanism
- SV-TrustEval-C: Evaluating Structure and Semantic Reasoning in Large Language Models for Source Code Vulnerability Analysis
- The Art of Repair: Optimizing Iterative Program Repair with Instruction-Tuned Models
- Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards
- ECLAIR: A Causally-Grounded AI Framework for Scientific Discovery in Empirical Software Engineering
- CockpitHAT: Dependency-Graph-Driven Hierarchical Attribution for Embodied Multi-Agent Cockpits
- From Reasoning to Generalization: Knowledge-Augmented LLMs for ARC Benchmark
- HyGenar: An LLM-Driven Hybrid Genetic Algorithm for Few-Shot Grammar Generation
- NL-Debugging: Exploiting Natural Language as an Intermediate Representation for Code Debugging
- R&D-Agent-Quant: A Multi-Agent Framework for Data-Centric Factors and Model Joint Optimization
- The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning
- Leveraging Large Language Models for Command Injection Vulnerability Analysis in Python: An Empirical Study on Popular Open-Source Projects
- Adversarial Reasoning for Repair Based on Inferred Program Intent
- From Automation to Autonomy: A Survey on Large Language Models in Scientific Discovery
- Intrinsic Self-Correction in LLMs: Towards Explainable Prompting via Mechanistic Interpretability
- SoftCoT++: Test-Time Scaling with Soft Chain-of-Thought Reasoning
- Mining Hidden Thoughts from Texts: Evaluating Continual Pretraining with Synthetic Data for LLM Reasoning
- Benchmarking LLMs on File System Design and Implementation
- MigrationBench: Repository-Level Code Migration Benchmark from Java 8
- CodePDE: An Inference Framework for LLM-driven PDE Solver Generation
- ToolACE-DEV: Self-Improving Tool Learning via Decomposition and EVolution
- Enhancing Code Generation via Bidirectional Comment-Level Mutual Grounding
- Benchmarking and Revisiting Code Generation Assessment: A Mutation-Based Approach
- When to Vote, When to Rewrite: Disagreement-Guided Strategy Routing for Test-Time Scaling
- Enhancing LLM Code Generation: A Systematic Evaluation of Multi-Agent Collaboration and Runtime Debugging for Improved Accuracy, Reliability, and Latency
- QiMeng-Xpiler: Transcompiling Tensor Programs for Deep Learning Systems with a Neural-Symbolic Approach
- Frontier Coding Agents Use Metaprogramming to Adapt to Unfamiliar Programming Languages
- VeriTrace: Human-Like Temporal Exploration Completes Agentic Action Space
- SciNav: A General Agent Framework for Scientific Coding Tasks
- ParEVO: Synthesizing Code for Irregular Data: High-Performance Parallelism through Agentic Evolution
- SciDER: Scientific Data-centric End-to-end Researcher
- Towards a Neural Debugger for Python
- LLM-Assisted Repository-Level Generation with Structured Spec-Driven Engineering
- kAgent: An execution-guided crash resolution agent for the Linux kernel
- A Systematic Literature Review of Parameter-Efficient Fine-Tuning for Large Code Models
- PaperClaw: Harnessing Agents for Autonomous Research and Human-in-the-Loop Refinement
- OpenSkill: Open-World Self-Evolution for LLM Agents
- Spec Kit Agents: Context-Grounded Agentic Workflows
- Compiler-Guided Inference-Time Adaptation: Improving GPT-5 Programming Performance in Idris
- VERITAS: A Multi-Agent Co-Scientist for Verifiable Image-Derived Hypothesis Testing
- The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break
- Route-Align-Verify for Functional Correctness in Code Generation
- Towards Adaptive Software Agents for Debugging
- The Order Is the Guarantee: Verifier-Budgeted Code Deletion with Static-First Learned Proposals
- Self-Adaptive Cognitive Debiasing for Large Language Models in Decision-Making
- APIRAT: Integrating Multi-source API Knowledge for Enhanced Code Translation with LLMs
- SQL-Factory: A Multi-Agent Framework for High-Quality and Large-Scale SQL Generation
- Think2SQL: Reinforce LLM Reasoning Capabilities for Text2SQL
- Data-efficient LLM Fine-tuning for Code Generation
- AdaCoder: An Adaptive Planning and Multi-Agent Framework for Function-Level Code Generation
- OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs
- Causal Episodic Memory for Feedback-Driven Agent Repair
- Reasoning from Traces: Divergence-Guided Agentic Repair of WebAssembly Discrepancies
- Evaluating and Improving Pedagogical Fit in LLM-Based AI Tutors with the Pedagogical Suitability Index
- From Guessing to Seeing: Enhancing LLM-Based Program Repair via Trace-Guided Multi-strategy Debate
- Efficient Reasoning Models: A Survey
- Abacus-SQL: A Text-to-SQL System Empowering Cross-Domain and Open-Domain Database Retrieval
- DocAgent: A Multi-Agent System for Automated Code Documentation Generation
- Synthesizing High-Quality Programming Tasks with LLM-based Expert and Student Agents
- FeedbackEval: A Benchmark for Evaluating Large Language Models in Feedback-Driven Code Repair Tasks
Discussions
Related