Training Language Models to Self-Correct via Reinforcement Learning

2024/09/19 by Aviral Kumar, Vincent Zhuang, Kumar, Aviral +36 · 2 voices · 76 citations
Computer Science · #Speech and dialogue systems #Topic Modeling #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2409.12917

Abstract

Self-correction is a highly desirable capability of large language models (LLMs), yet it has consistently been found to be largely ineffective in modern LLMs. Current methods for training self-correction typically depend on either multiple models, a more advanced model, or additional forms of supervision. To address these shortcomings, we develop a multi-turn online reinforcement learning (RL) approach, SCoRe, that significantly improves an LLM's self-correction ability using entirely self-generated data. To build SCoRe, we first show that variants of supervised fine-tuning (SFT) on offline model-generated correction traces are often insufficient for instilling self-correction behavior. In particular, we observe that training via SFT falls prey to either a distribution mismatch between mistakes made by the data-collection policy and the model's own responses, or to behavior collapse, where learning implicitly prefers only a certain mode of correction behavior that is often not effective at self-correction on test problems. SCoRe addresses these challenges by training under the model's own distribution of self-generated correction traces and using appropriate regularization to steer the learning process into learning a self-correction behavior that is effective at test time as opposed to fitting high-reward responses for a given prompt. This regularization process includes an initial phase of multi-turn RL on a base model to generate a policy initialization that is less susceptible to collapse, followed by using a reward bonus to amplify self-correction. With Gemini 1.0 Pro and 1.5 Flash models, we find that SCoRe achieves state-of-the-art self-correction performance, improving the base models' self-correction by 15.6% and 9.1% respectively on MATH and HumanEval.

Cited by

• LeAct: Learning to Reason from Expert Actions
• Artificial Intelligence for Mathematical Reasoning: An Integrated Survey of Language Models, Neuro-symbolic Systems, and Verified Discovery
• Can We Break LLMs Out of Self-Loops? Fine-Grained Reasoning Control with Activation Steering
• SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning
• Meta-Reinforcement Learning with Self-Reflection for Agentic Search
• Bootstrapping Task Spaces for Self-Improvement
• The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs
• Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs
• Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
• StAR: Segment Anything Reasoner
• Scalable Stewardship of an LLM-Assisted Clinical Benchmark with Physician Oversight
• A Theoretical Lens for RL-Tuned Language Models via Energy-Based Models
• Meta-RL Induces Exploration in Language Agents
• Stackelberg Learning from Human Feedback: Preference Optimization as a Sequential Game
• Toward More Reliable Artificial Intelligence: Reducing Hallucinations in Vision-Language Models
• Think-Reflect-Revise: A Policy-Guided Reflective Framework for Safety Alignment in Large Vision Language Models
• RefineBench: Evaluating Refinement Capability of Language Models via Checklists
• Natural Language Actor-Critic: Scalable Off-Policy Learning in Language Space
• Synthetic Error Injection Fails to Elicit Self-Correction In Language Models
• On the Tension Between Optimality and Adversarial Robustness in Policy Optimization
• Probing the "Psyche'' of Large Reasoning Models: Understanding Through a Human Lens
• REVISOR: Beyond Textual Reflection, Towards Multimodal Introspective Reasoning in Long-Form Video Understanding
• Transformers with RL or SFT Provably Learn Sparse Boolean Functions, But Differently
• From Solving to Verifying: A Unified Objective for Robust Reasoning in LLMs
• GRAPH-GRPO-LEX: Contract Graph Modeling and Reinforcement Learning with Group Relative Policy Optimization
• SymCode: A Neurosymbolic Approach to Mathematical Reasoning via Verifiable Code Generation
• LANPO: Bootstrapping Language and Numerical Feedback for Reinforcement Learning in LLMs
• Parrot: A Training Pipeline Enhances Both Program CoT and Natural Language CoT for Reasoning
• The Narrative Continuity Test: A Conceptual Framework for Evaluating Identity Persistence in AI Systems
• PACR: Progressively Ascending Confidence Reward for LLM Reasoning
• Boosting Accuracy and Efficiency of Budget Forcing in LLMs via Reinforcement Learning for Mathematical Reasoning
• Multi-turn Training with Basic Human Feedback Helps Little on LLM Reasoning
• Deep Self-Evolving Reasoning
• Generative Universal Verifier as Multimodal Meta-Reasoner
• Self-Verifying Reflection Helps Transformers with CoT Reasoning
• Self-Forcing++: Towards Minute-Scale High-Quality Video Generation
• Follow My Lead: Logical Fallacy Classification with Knowledge-Augmented LLMs
• Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey
• Can RL Improve Generalization of LLM Agents? An Empirical Study
• SaFeR-VLM: Toward Safety-aware Fine-grained Reasoning in Multimodal Models
• XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitation
• Optimizing for Persuasion Improves LLM Generalization: Evidence from Quality-Diversity Evolution of Debate Strategies
• Fine-Tuning Masked Diffusion for Provable Self-Correction
• RLAD: Training LLMs to Discover Abstractions for Solving Reasoning Problems
• Detecting Distillation Data from Reasoning Models
• Unlocking Reasoning Capabilities in LLMs via Reinforcement Learning Exploration
• GuidedSampling: Steering LLMs Towards Diverse Candidate Solutions at Inference-Time
• Self-Reflective Generation at Test Time
• Latent Thinking Optimization: Your Latent Reasoning Language Model Secretly Encodes Reward Signals in Its Latent Thoughts
• Scalable and Robust LLM Unlearning by Correcting Responses with Retrieved Exclusions
• CDT: A Comprehensive Capability Framework for Large Language Models Across Cognition, Domain, and Task
• Risk-Sensitive RL for Alleviating Exploration Dilemmas in Large Language Models
• Unlocking Zero-Shot Geospatial Reasoning via Indirect Rewards
• Poivre: Self-Refining Visual Pointing with Reinforcement Learning
• Uncovering Vulnerabilities of LLM-Assisted Cyber Threat Intelligence
• EAPO: Enhancing Policy Optimization with On-Demand Expert Assistance
• From Harm to Help: Turning Reasoning In-Context Demos into Assets for Reasoning LMs
• Test-Time Policy Adaptation for Enhanced Multi-Turn Interactions with LLMs
• A2R: An Asymmetric Two-Stage Reasoning Framework for Parallel Reasoning
• Variation in Verification: Understanding Verification Dynamics in Large Language Models
• DeepDive: Advancing Deep Search Agents with Knowledge Graphs and Multi-Turn RL
• Abduct, Act, Predict: Scaffolding Causal Inference for Automated Failure Attribution in Multi-Agent Systems
• Generative Data Refinement: Just Ask for Better Data
• AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning
• Beyond Memorization: Extending Reasoning Depth with Recurrence, Memory and Test-Time Compute Scaling
• Reverse-Engineered Reasoning for Open-Ended Generation
• A Probabilistic Inference Scaling Theory for LLM Self-Correction
• Learning When to Plan: Efficiently Allocating Test-Time Compute for LLM Agents
• Text Reinforcement for Multimodal Time Series Forecasting
• MM-SeR: Multimodal Self-Refinement for Lightweight Image Captioning
• A Stitch in Time Saves Nine: Proactive Self-Refinement for Language Models
• ThinkTuning: Instilling Cognitive Reflections without Distillation
• Towards Theoretical Understanding of Transformer Test-Time Computing: Investigation on In-Context Linear Regression
• Uncertainty-Aware Semantic Decoding for LLM-Based Sequential Recommendation
• SC-Captioner: Improving Image Captioning with Self-Correction by Reinforcement Learning
• Self-Questioning Language Models

Discussions

• Training Language Models to Self-Correct via Reinforcement Learning [hn, 230 points, 92 comments]
• Training Language Models to Self-Correct via Reinforcement Learning https://arxiv.org/abs/2409.12917 [bsky, 0 points, 0 comments]

Related

• Self-Correction Bench: Uncovering and Addressing the Self-Correction Blind Spot in Large Language Models
• Self-Correcting Code Generation Using Small Language Models
• Synthetic Error Injection Fails to Elicit Self-Correction In Language Models
• Post-Training Large Language Models via Reinforcement Learning from Self-Feedback
• Confidence v.s. Critique: A Decomposition of Self-Correction Capability for LLMs