A Theoretical Understanding of Self-Correction through In-context Alignment

2024/05/28 by Yifei Wang, Yuyang Wu, Wang, Yifei +7 · 24 citations
Social Sciences · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Multimedia Communication and Technology

paper · pdf · doi:10.48550/arxiv.2405.18634

openalex publication_date 2024/05/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Going beyond mimicking limited human experiences, recent studies show initial evidence that, like humans, large language models (LLMs) are capable of improving their abilities purely by self-correction, i.e., correcting previous responses through self-examination, in certain circumstances. Nevertheless, little is known about how such capabilities arise. In this work, based on a simplified setup akin to an alignment task, we theoretically analyze self-correction from an in-context learning perspective, showing that when LLMs give relatively accurate self-examinations as rewards, they are capable of refining responses in an in-context way. Notably, going beyond previous theories on over-simplified linear transformers, our theoretical construction underpins the roles of several key designs of realistic transformers for self-correction: softmax attention, multi-head attention, and the MLP block. We validate these findings extensively on synthetic datasets. Inspired by these findings, we also illustrate novel applications of self-correction, such as defending against LLM jailbreaks, where a simple self-correction step does make a large difference. We believe that these findings will inspire further research on understanding, exploiting, and enhancing self-correction for building better foundation models.

Cited by

• LANPO: Bootstrapping Language and Numerical Feedback for Reinforcement Learning in LLMs
• DynaQuery: A Self-Adapting Framework for Querying Structured and Multimodal Data
• CRepair Wrapper Improves Structural Self-Repair Across Three LLM Families: A Cross-Model Replication Study
• MetaDefense: Defending Finetuning-based Jailbreak Attack Before and During Generation
• Scalable and Robust LLM Unlearning by Correcting Responses with Retrieved Exclusions
• SAC-Opt: Semantic Anchors for Iterative Correction in Optimization Modeling
• Reinforced Visual Perception with Tools
• An Explainable Emotion Alignment Framework for LLM-Empowered Agent in Metaverse Service Ecosystem
• Specification Self-Correction: Mitigating In-Context Reward Hacking Through Test-Time Refinement
• CIMR: Contextualized Iterative Multimodal Reasoning for Robust Instruction Following in LVLMs
• Rethinking Invariance in In-context Learning
• Discourse Heuristics For Paradoxically Moral Self-Correction
• Theoretical Modeling of Large Language Model Self-Improvement Training Dynamics Through Solver-Verifier Gap
• When and How Unlabeled Data Provably Improve In-Context Learning
• HEAL: An Empirical Study on Hallucinations in Embodied Agents Driven by Large Language Models
• Long-Short Alignment for Effective Long-Context Modeling in LLMs
• ReGA: Representation-Guided Abstraction for Model-based Safeguarding of LLMs
• Pretraining Language Models to Ponder in Continuous Space
• Self-Reflective Planning with Knowledge Graphs: Enhancing LLM Reasoning Reliability for Question Answering
• Large Language Models for Planning: A Comprehensive and Systematic Survey
• Scalable Defense against In-the-wild Jailbreaking Attacks with Safety Context Retrieval
• Intrinsic Self-Correction in LLMs: Towards Explainable Prompting via Mechanistic Interpretability
• Follow the Path: Reasoning over Knowledge Graph Paths to Improve Large Language Model Factuality
• Finding and Reactivating Post-Trained LLMs' Hidden Safety Mechanisms

Related

• Large Language Models have Intrinsic Self-Correction Ability
• Confidence v.s. Critique: A Decomposition of Self-Correction Capability for LLMs
• Training Language Models to Self-Correct via Reinforcement Learning
• On the Intrinsic Self-Correction Capability of LLMs: Uncertainty and Latent Concept
• Confidence Matters: Revisiting Intrinsic Self-Correction Capabilities of Large Language Models