Teaching Large Language Models to Self-Debug
2023/04/11 by Xinyun Chen, Maxwell Lin, Chen, Xinyun +5 · 2 voices · 135 citations
Computer Science · #Software Engineering Research #Parallel Computing and Optimization Techniques #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2304.05128
Abstract
Large language models (LLMs) have achieved impressive performance on code generation. However, for complex programming tasks, generating the correct solution in one go becomes challenging, thus some prior works have designed program repair approaches to improve code generation performance. In this work, we propose Self-Debugging, which teaches a large language model to debug its predicted program via few-shot demonstrations. In particular, we demonstrate that Self-Debugging can teach the large language model to perform rubber duck debugging; i.e., without any human feedback on the code correctness or error messages, the model is able to identify its mistakes by investigating the execution results and explaining the generated code in natural language. Self-Debugging achieves the state-of-the-art performance on several code generation benchmarks, including the Spider dataset for text-to-SQL generation, TransCoder for C++-to-Python translation, and MBPP for text-to-Python generation. On the Spider benchmark where there are no unit tests to verify the correctness of predictions, Self-Debugging with code explanation consistently improves the baseline by 2-3%, and improves the prediction accuracy on problems of the hardest level by 9%. On TransCoder and MBPP where unit tests are available, Self-Debugging improves the baseline accuracy by up to 12%. Meanwhile, by leveraging feedback messages and reusing failed predictions, Self-Debugging notably improves sample efficiency, and can match or outperform baseline models that generate more than 10x candidate programs.
Cited by
- Beyond Text-to-SQL: Can LLMs Really Debug Enterprise ETL SQL?
- Structured Feedback Improves Repair in an LLM Agent Loop
- Cross-Model LLM Code Review: Should you use Claude to review Codex or vice versa?
- Bridging the Last Mile of Circuit Design: PostEDA-Bench, a Hierarchical Benchmark for PPA Convergence and DRC Fixing
- CodeRescue: Budget-Calibrated Recovery Routing for Coding Agents
- DataFlow-Harness: A Grounded Code-Agent Platform for Constructing Editable LLM Data Pipelines
- Fully Automated End-to-End Adversary Emulation from MITRE ATT&CK Based Cyber Threat Intelligence Using LLMs
- CIPHER: A Decoupled Exploration-Selection Framework for Test-Time Scaling of Data Science Agents
- Models Can Model, But Can't Bind: Structured Grounding in Text-to-Optimization
- ExecuGraph: A Multi-Agent, Execution-Grounded Framework for Reliable Backend Code Synthesis with Large Language Models
- RAIL Guard: Closing the Evaluation-to-Remediation Gap in Responsible AI for LLM Agents
- VeriSimpl: Robust Optimization Modeling from Natural Language using Simplification-based Verification
- FormulaSPIN: Self-Play Fine-Tuning for Natural Language to Spreadsheet Formula Generation
- LLM-Based Agentic Systems for Software Engineering: Challenges and Opportunities
- Reasoning Models Reason Well, Until They Don't
- EditLord: Learning Code Transformation Rules for Code Editing
- Gold-medalist Performance in Solving Olympiad Geometry with AlphaGeometry2
- Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges
- NERFIFY: A Multi-Agent Framework for Turning NeRF Papers into Code
- DualityCert: Verifier-Gated Language-Model Repair of Broken Duality Claims in Quantum Field Theory
- Multi-level Code Optimization via Mixture of Prompts
- Specula: Scaling formal specifications for autonomous model checking of system code
- Beyond "What to Retrieve": Uncertainty in Retrieval-Augmented Code Generation
- Procedural Knowledge at Scale Improves Reasoning
- AutoBaxBuilder: Bootstrapping Code Security Benchmarking
- VIGOR+: Iterative Confounder Generation and Validation via LLM-CEVAE Feedback Loop
- SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios
- SoK: Understanding (New) Security Issues Across AI4Code Use Cases
- A Network Arena for Benchmarking AI Agents on Network Troubleshooting
- FutureWeaver: Planning Test-Time Compute for Multi-Agent Systems with Modularized Collaboration
- CogMCTS: A Novel Cognitive-Guided Monte Carlo Tree Search Framework for Iterative Heuristic Evolution with Large Language Models
- VIGIL: A Reflective Runtime for Self-Healing Agents
- Mitigating Self-Preference by Authorship Obfuscation
- RefineBench: Evaluating Refinement Capability of Language Models via Checklists
- On the Limits of Test-Time Compute: Sequential Reward Filtering for Better Inference
- SAGE: Semantic-Aware Gray-Box Game Regression Testing with Large Language Models
- Generating Verifiable Chain of Thoughts from Exection-Traces
- EWE: An Agentic Framework for Extreme Weather Analysis
- Progress by Pieces: Test-Time Scaling for Autoregressive Image Generation
- LLMs as Firmware Experts: A Runtime-Grown Tree-of-Agents Framework
- ELPO: Ensemble Learning Based Prompt Optimization for Large Language Models
- SDA: Steering-Driven Distribution Alignment for Open LLMs without Fine-Tuning
- InfCode: Adversarial Iterative Refinement of Tests and Patches for Reliable Software Issue Resolution
- From Solving to Verifying: A Unified Objective for Robust Reasoning in LLMs
- Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?
- Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talk
- RulePilot: An LLM-Powered Agent for Security Rule Generation
- ExPairT-LLM: Exact Learning for LLM Code Selection by Pairwise Queries
- ACT as Human: Multimodal Large Language Model Data Annotation with Critical Thinking
- MACEval: A Multi-Agent Continual Evaluation Network for Large Models
- Self-Correction Distillation for Structured Data Question Answering
- Retriv at BLP-2025 Task 2: Test-Driven Feedback-Guided Framework for Bangla-to-Python Code Generation
- VeriCoT: Neuro-symbolic Chain-of-Thought Validation via Logical Consistency Checks
- EDIT-Bench: Evaluating LLM Abilities to Perform Real-World Instructed Code Edits
- ReAcTree: Hierarchical LLM Agent Trees with Control Flow for Long-Horizon Task Planning
- The Sequential Edge: Inverse-Entropy Voting Beats Parallel Self-Consistency at Matched Compute
- HarnessLLM: Automatic Testing Harness Generation via Reinforcement Learning
- ORANGE: An Online Reflection ANd GEneration framework with Domain Knowledge for Text-to-SQL
- PDE-SHARP: PDE Solver Hybrids through Analysis and Refinement Passes
- SELF-REDRAFT: Eliciting Intrinsic Exploration-Exploitation Balance in Test-Time Scaling for Code Generation
- Generalizing Test-time Compute-optimal Scaling as an Optimizable Graph
- Debate2Create: Robot Co-design via Large Language Model Debates
- From Role Prompt to Infinite Thinking: Exploiting Persona Conditioning for Inference Cost Attacks in LLMs
- Do Code Language Models Use Tests? A Behavioral and Representational Study of Test-Driven Code Generation
- ClinLens: Towards Long-Horizon Coding Agents for Longitudinal Multimodal Clinical Data Science
- TraceCoder: Explainable and Auditable Code Generation with Position-Key Snippet Versioning
- EsoLang-Bench: Evaluating Genuine Reasoning in Large Language Models via Esoteric Programming Languages
- Lang-PINN: From Language to Physics-Informed Neural Networks via a Multi-Agent Framework
- Flows: Building Blocks of Reasoning and Collaborating AI
- TEXT2DB: Integration-Aware Information Extraction with Large Language Model Agents
- Once Upon an Input: Reasoning via Per-Instance Program Synthesis
- Software Engineering Agents for Embodied Controller Generation : A Study in Minigrid Environments
- Finding the Sweet Spot: Trading Quality, Cost, and Speed During Inference-Time LLM Reflection
- Learning from Supervision with Semantic and Episodic Memory: A Reflective Approach to Agent Adaptation
- Knowledge-Guided Multi-Agent Framework for Application-Level Software Code Generation
- Illusions of reflection: open-ended task reveals systematic failures in Large Language Models' reflective reasoning
- Enhancing geodatabases operability: advanced human-computer interaction through RAG and Multi-Agent Systems
- SpecAgent: A Speculative Retrieval and Forecasting Agent for Code Completion
- An Agentic Framework with LLMs for Solving Complex Vehicle Routing Problems
- LSPO: Length-aware Dynamic Sampling for Policy Optimization in LLM Reasoning
- David vs. Goliath: A comparative study of different-sized LLMs for code generation in the domain of automotive scenario generation
- EvoCAD: Evolutionary CAD Code Generation with Vision Language Models
- DebugTA: An LLM-Based Agent for Simplifying Debugging and Teaching in Programming Education
- Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey
- MUSE-Autoskill: Self-Evolving Agents via Skill Creation, Memory, Management, and Evaluation
- Autonomous Agents for Scientific Discovery: Orchestrating Scientists, Language, Code, and Physics
- Fundamentals of Building Autonomous LLM Agents
- Self-Improving LLM Agents at Test-Time
- Traceability and Accountability in Role-Specialized Multi-Agent LLM Pipelines
- XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitation
- RECODE-H: A Benchmark for Research Code Development with Interactive Human Feedback
- AlphaApollo: Orchestrating Foundation Models and Professional Tools into a Self-Evolving System for Deep Agentic Reasoning
- A Survey of LLM-Based Applications in Programming Education: Balancing Automation and Human Oversight
- A Tale of LLMs and Induced Small Proxies: Scalable Small Language Models for Knowledge Mining
- LLM-Based Multi-Agent Blackboard System for Information Discovery in Data Science
- Towards Reliable Generation of Executable Workflows by Foundation Models
- Multilingual Text-to-SQL: Benchmarking the Limits of Language Models with Collaborative Language Agents
- LLM/Agent-as-Data-Analyst: A Survey
- Large-Scale Constraint Generation -- Can LLMs Parse Hundreds of Constraints?
- From Harm to Help: Turning Reasoning In-Context Demos into Assets for Reasoning LMs
- RepLLM: Toward Automatically Reproducing Network Research Results
- RefineSVG: Visual Feedback-Driven Reinforcement Learning for Image-to-SVG Generation
- SCOPE: Synthetic Conditional Objectives for Policy Evolution in Black-Box Combinatorial Optimization
- Scaling LLM-Driven Multi-Agent Systems: Design Principles and Architectural Scalability Analysis
- Code Driven Planning with Domain-Adaptive Critic
- Actions Speak Louder than Prompts: A Large-Scale Study of LLMs for Graph Inference
- Variation in Verification: Understanding Verification Dynamics in Large Language Models
- Building Data-Driven Occupation Taxonomies: A Bottom-Up Multi-Stage Approach via Semantic Clustering and Multi-Agent Collaboration
- Generating High-Quality Datasets for Code Editing via Open-Source Language Models
- DF-LLaVA: Unlocking MLLMs for Synthetic Image Detection via Knowledge Injection and Conflict-Driven Self-Reflection
- An Empirical Study on Failures in Automated Issue Solving
- VerilogMonkey: Exploring Parallel Scaling for Automated Verilog Code Generation with LLMs
- FVDebug: An LLM-Driven Debugging Assistant for Automated Root Cause Analysis of Formal Verification Failures
- LTA-thinker: Latent Thought-Augmented Training Framework for Large Language Models on Complex Reasoning
- CodeCureAgent: Automatic Classification and Repair of Static Analysis Warnings
- Do Code Semantics Help? A Comprehensive Study on Execution Trace-Based Information for Code Large Language Models
- SQLGovernor: An LLM-powered SQL Toolkit for Real World Application
- PaVeRL-SQL: Text-to-SQL via Partial-Match Rewards and Verbal Reinforcement Learning
- A Probabilistic Inference Scaling Theory for LLM Self-Correction
- ReCode: Improving LLM-based Code Repair with Fine-Grained Retrieval-Augmented Generation
- Aligning Requirement for Large Language Model's Code Generation
- Can Multi-turn Self-refined Single Agent LMs with Retrieval Solve Hard Coding Problems?
- Ensemble Debates with Local Large Language Models for AI Alignment
- A Stitch in Time Saves Nine: Proactive Self-Refinement for Language Models
- Hell or High Water: Evaluating Agentic Recovery from External Failures
- LLM driven Text-to-Table Generation through Sub-Tasks Guidance and Iterative Refinement
- Uncertainty-Aware Semantic Decoding for LLM-Based Sequential Recommendation
- GP and LLMs for Program Synthesis: No Clear Winners
- Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-Correction
- MRG-Bench: Evaluating and Exploring the Requirements of Context for Repository-Level Code Generation
- A Rolling Stone Gathers No Moss: Adaptive Policy Optimization for Stable Self-Evaluation in Large Multimodal Models
- HyCodePolicy: Hybrid Language Controllers for Multimodal Monitoring and Decision in Embodied Agents
- Exploring Direct Instruction and Summary-Mediated Prompting in LLM-Assisted Code Modification
- WarriorMath: Enhancing the Mathematical Ability of Large Language Models with a Defect-aware Framework
- Chatting with your ERP: A Recipe
Discussions
Related