Large Language Models Can Self-Improve

2022/10/20 by Jiaxin Huang, Shixiang Gu, Shixiang Shane Gu +12 · 3 voices · 81 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.2210.11610

openalex publication_date 2022/10/20 · arxiv published 2022/10/20 · arxiv updated 2022/10/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) have achieved excellent performances in various tasks. However, fine-tuning an LLM requires extensive supervision. Human, on the other hand, may improve their reasoning abilities by self-thinking without external inputs. In this work, we demonstrate that an LLM is also capable of self-improving with only unlabeled datasets. We use a pre-trained LLM to generate "high-confidence" rationale-augmented answers for unlabeled questions using Chain-of-Thought prompting and self-consistency, and fine-tune the LLM using those self-generated solutions as target outputs. We show that our approach improves the general reasoning ability of a 540B-parameter LLM (74.4%->82.1% on GSM8K, 78.2%->83.0% on DROP, 90.0%->94.4% on OpenBookQA, and 63.4%->67.9% on ANLI-A3) and achieves state-of-the-art-level performance, without any ground truth label. We conduct ablation studies and show that fine-tuning on reasoning is critical for self-improvement.

Cited by

• Fine-Tuning LLMs with Fine-Grained Human Feedback on Text Spans
• SERM: Self-Evolving Relevance Model with Agent-Driven Learning from Massive Query Streams
• From Shallow Humor to Metaphor: Towards Label-Free Harmful Meme Detection via LMM Agent Self-Improvement
• CienaLLM: Generative Climate-Impact Extraction from News Articles with Autoregressive LLMs
• Intelligent Human-Machine Partnership for Manufacturing: Enhancing Warehouse Planning through Simulation-Driven Knowledge Graphs and LLM Collaboration
• Differences That Matter: Auditing Models for Capability Gap Discovery and Rectification
• SpeakRL: Synergizing Reasoning, Speaking, and Acting in Language Models with Reinforcement Learning
• GLaD: Geometric Latent Distillation for Vision-Language-Action Models
• To Think or Not to Think: The Hidden Cost of Meta-Training with Excessive CoT Examples
• rSIM: Incentivizing Reasoning Capabilities of LLMs via Reinforced Strategy Injection
• Evolutionary System 2 Reasoning: An Empirical Proof
• The Road of Adaptive AI for Precision in Cybersecurity
• Decentralized Multi-Agent System with Trust-Aware Communication
• Differential Smoothing Mitigates Sharpening and Improves LLM Reasoning
• Cross-Disciplinary Knowledge Retrieval and Synthesis: A Compound AI Architecture for Scientific Discovery
• LLMs as Firmware Experts: A Runtime-Grown Tree-of-Agents Framework
• SPINE: Token-Selective Test-Time Reinforcement Learning with Entropy-Band Regularization
• Do Vision-Language Models Understand Visual Persuasiveness?
• Incorporating Self-Rewriting into Large Language Model Reasoning Reinforcement
• RB-FT: Rationale-Bootstrapped Fine-Tuning for Video Classification
• SVBRD-LLM: Self-Verifying Behavioral Rule Discovery for Autonomous Vehicle Identification
• AutoMalDesc: Large-Scale Script Analysis for Cyber Threat Research
• ACT as Human: Multimodal Large Language Model Data Annotation with Critical Thinking
• MathSE: Improving Multimodal Mathematical Reasoning via Self-Evolving Iterative Reflection and Reward-Guided Fine-Tuning
• Self-Harmony: Learning to Harmonize Self-Supervision and Self-Play in Test-Time Reinforcement Learning
• Evontree: Ontology Rule-Guided Self-Evolution of Large Language Models
• Counteracting Matthew Effect in Self-Improvement of LVLMs through Head-Tail Re-balancing
• Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning
• OncoReason: Structuring Clinical Reasoning in LLMs for Robust and Interpretable Survival Prediction
• Video Reasoning without Training
• Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMs
• Flip-Flop Consistency: Unsupervised Training for Robustness to Prompt Perturbations in LLMs
• Stop Reducing Responsibility in LLM-Powered Multi-Agent Systems to Local Alignment
• Training LLM Agents to Empower Humans
• Towards Human-Centric Intelligent Treatment Planning for Radiation Therapy
• KnowRL: Teaching Language Models to Know What They Know
• BrowserAgent: Building Web Agents with Human-Inspired Web Browsing Actions
• LLM-Friendly Knowledge Representation for Customer Support
• Agent Learning via Early Experience
• Self-Improving LLM Agents at Test-Time
• Do LLMs Know They Are Being Tested? Evaluation Awareness and Incentive-Sensitive Failures in GPT-OSS-20B
• Self-Filtered Distillation with LLMs-generated Trust Indicators for Reliable Patent Classification
• Large Language Models Hallucination: A Comprehensive Survey
• Round-trip Reinforcement Learning: Self-Consistent Training for Better Chemical LLMs
• Self-Improvement in Multimodal Large Language Models: A Survey
• Towards Trustworthy Lexical Simplification: Exploring Safety and Efficiency with Small LLMs
• Sanitize Your Responses: Mitigating Privacy Leakage in Large Language Models
• Semantic Voting: A Self-Evaluation-Free Approach for Efficient LLM Self-Improvement on Unverifiable Open-ended Tasks
• ReviewScore: Misinformed Peer Review Detection with Large Language Models
• RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs
• MARS: toward more efficient multi-agent collaboration for LLM reasoning
• ToolBrain: A Flexible Reinforcement Learning Framework for Agentic Tools
• EnAnchored-X2X: English-Anchored Optimization for Many-to-Many Translation
• HSS-Synth: Humanities and Social Sciences Data Synthesis for LLMs
• RELISH: LLM REgression with a Latent Iterative State Head
• Unveiling Chain of Step Reasoning for Vision-Language Models with Fine-grained Rewards
• Variation in Verification: Understanding Verification Dynamics in Large Language Models
• Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variation
• The Prompt Engineering Report Distilled: Quick Start Guide for Life Sciences
• Mitigating Hallucinations in Large Vision-Language Models by Self-Injecting Hallucinations
• VARCO-VISION-2.0 Technical Report
• Evaluating LLMs Without Oracle Feedback: Agentic Annotation Evaluation Through Unsupervised Consistency Signals
• Language Self-Play For Data-Free Training
• Mitigating Spurious Correlations Between Question and Answer via Chain-of-Thought Correctness Perception Distillation
• CoVeR: Conformal Calibration for Versatile and Reliable Autoregressive Next-Token Prediction
• TCIA: A Task-Centric Instruction Augmentation Method for Instruction Finetuning
• Entropy-Guided Loop: Achieving Reasoning through Uncertainty-Aware Generation
• LENS: Learning to Segment Anything with Unified Reinforced Reasoning
• Wisdom of the Crowd: Reinforcement Learning from Coevolutionary Collective Feedback
• Efficient Switchable Safety Control in LLMs via Magic-Token-Guided Co-Training
• MolmoAct: Action Reasoning Models that can Reason in Space
• Assessing and Mitigating Data Memorization Risks in Fine-Tuned Large Language Models
• Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Future
• Boosting Visual Knowledge-Intensive Training for LVLMs Through Causality-Driven Visual Object Completion
• Method-Based Reasoning for Large Language Models: Extraction, Reuse, and Continuous Improvement
• ReasonAct: Progressive Training for Fine-Grained Video Reasoning in Small Models
• Kimi K2: Open Agentic Intelligence
• Self-Improvement for Audio Large Language Model using Unlabeled Speech
• Efficient Learning for Product Attributes with Compact Multimodal Models
• Maximizing Prefix-Confidence at Test-Time Efficiently Improves Mathematical Reasoning
• Multimodal Behavioral Patterns Analysis with Eye-Tracking and LLM-Based Reasoning

Discussions

• Large Language Models Can Self-Improve [hn, 3 points, 0 comments]
• Large language models can self-improve [hn, 3 points, 0 comments]
• Large Language Models Can Self-Improve [hn, 2 points, 0 comments]

Related

• Self-Training Large Language Models with Confident Reasoning
• The Self-Improvement Paradox: Can Language Models Bootstrap Reasoning Capabilities without External Scaffolding?
• Large Language Models Cannot Self-Correct Reasoning Yet
• Self-Training Elicits Concise Reasoning in Large Language Models
• Self-Reasoning Language Models: Unfold Hidden Reasoning Chains with Few Reasoning Catalyst