Self-Improvement in Language Models: The Sharpening Mechanism

2024/12/02 by Audrey Huang, Adam Block, Huang, Audrey +13 · 2 voices · 24 citations
Computer Science · #Natural Language Processing Techniques #Topic Modeling #cs.AI #cs.CL #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2412.01951

openalex publication_date 2024/12/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent work in language modeling has raised the possibility of self-improvement, where a language models evaluates and refines its own generations to achieve higher performance without external feedback. It is impossible for this self-improvement to create information that is not already in the model, so why should we expect that this will lead to improved capabilities? We offer a new perspective on the capabilities of self-improvement through a lens we refer to as sharpening. Motivated by the observation that language models are often better at verifying response quality than they are at generating correct responses, we formalize self-improvement as using the model itself as a verifier during post-training in order to ``sharpen'' the model to one placing large mass on high-quality sequences, thereby amortizing the expensive inference-time computation of generating good sequences. We begin by introducing a new statistical framework for sharpening in which the learner aims to sharpen a pre-trained base policy via sample access, and establish fundamental limits. Then we analyze two natural families of self-improvement algorithms based on SFT and RLHF. We find that (i) the SFT-based approach is minimax optimal whenever the initial model has sufficient coverage, but (ii) the RLHF-based approach can improve over SFT-based self-improvement by leveraging online exploration, bypassing the need for coverage. Finally, we empirically validate the sharpening mechanism via inference-time and amortization experiments. We view these findings as a starting point toward a foundational understanding that can guide the design and evaluation of self-improvement algorithms.

Cited by

• Self-Adapting Language Models
• Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges
• SpeakRL: Synergizing Reasoning, Speaking, and Acting in Language Models with Reinforcement Learning
• MarkTune: Improving the Quality-Detectability Trade-off in Open-Weight LLM Watermarking
• When Does Verification Pay Off? A Closer Look at LLMs as Solution Verifiers
• How Does RL Post-training Induce Skill Composition? A Case Study on Countdown
• Mode-Conditioning Unlocks Superior Test-Time Scaling
• Towards Active Synthetic Data Generation for Finetuning Language Models
• Differential Smoothing Mitigates Sharpening and Improves LLM Reasoning
• A Representation Sharpening Framework for Zero Shot Dense Retrieval
• KL-Regularized Reinforcement Learning is Designed to Mode Collapse
• A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning
• Training LLM Agents to Empower Humans
• Representation-Based Exploration for Language Models: From Test-Time to Post-Training
• Self-Improving LLM Agents at Test-Time
• Learning on the Job: Test-Time Curricula for Targeted Reinforcement Learning
• Best-of-Majority: Minimax-Optimal Strategy for Pass@k Inference Scaling
• Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewards
• Evaluating large language models on business process modeling: framework, benchmark, and self-improvement analysis
• RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs
• Jointly Reinforcing Diversity and Quality in Language Model Generations
• Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Future
• Large Language Models Facilitate Vision Reflection in Image Classification
• Post-Completion Learning for Language Models

Discussions

• Given a high-quality verifier, language model accuracy can be improved by scaling inference-time compute (e.g., w/ repeated sampling). When can we expect similar gains without an external verifier? N [bsky, 41 points, 3 comments]
• Can a language model improve itself without external verifier? We pose self-improvement as a computational challenge, and show how self-training might surmount it. Joint work with @djfoster.bsky.socia [bsky, 2 points, 1 comments]

Related

• Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models
• Teaching Language Models to Self-Improve by Learning from Language Feedback
• SELF: Self-Evolution with Language Feedback
• Latent Principle Discovery for Language Model Self-Improvement
• Self-Refine: Iterative Refinement with Self-Feedback