Self-Rewarding Language Models

2024/01/18 by Weizhe Yuan, Richard Yuanzhe Pang, Yuan, Weizhe +11 · 4 voices · 167 citations
Computer Science · Psychology · #Computer science #Natural Language Processing Techniques #Psychology #Topic Modeling #cs.AI #cs.CL

paper · pdf · doi:10.48550/arxiv.2401.10020

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/01/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We posit that to achieve superhuman agents, future models require superhuman feedback in order to provide an adequate training signal. Current approaches commonly train reward models from human preferences, which may then be bottlenecked by human performance level, and secondly these separate frozen reward models cannot then learn to improve during LLM training. In this work, we study Self-Rewarding Language Models, where the language model itself is used via LLM-as-a-Judge prompting to provide its own rewards during training. We show that during Iterative DPO training that not only does instruction following ability improve, but also the ability to provide high-quality rewards to itself. Fine-tuning Llama 2 70B on three iterations of our approach yields a model that outperforms many existing systems on the AlpacaEval 2.0 leaderboard, including Claude 2, Gemini Pro, and GPT-4 0613. While there is much left still to explore, this work opens the door to the possibility of models that can continually improve in both axes.

Cited by

• Teaching LLMs to Self-Evolve: Cultivating Core Meta-Skills with Reinforcement Learning
• Dr. Zero: Self-Evolving Search Agents without Training Data
• FlowEvo: Self-Evolving Agents through the Co-Evolution of Workflows and Executable Skills
• Embarrassingly Simple Self-Distillation Improves Code Generation
• Distributional AGI Safety
• RLSR: Reinforcement Learning from Self Reward
• Absolute Zero: Reinforced Self-play Reasoning with Zero Data
• Inference-Time Scaling for Generalist Reward Modeling
• Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges
• Self-Rewarded Multimodal Coherent Reasoning Across Diverse Visual Domains
• From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement
• Self-Boosting Vision-Language Models with Noisy Student On-Policy Self-Distillation
• In-Context Learning as Implicit Policy Gradient
• Search Beyond What Can Be Taught: Evolving the Knowledge Boundary in Agentic Visual Generation
• Differences That Matter: Auditing Models for Capability Gap Discovery and Rectification
• Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learning
• Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Image
• TraPO: A Semi-Supervised Reinforcement Learning Framework for Boosting LLM Reasoning
• GLaD: Geometric Latent Distillation for Vision-Language-Action Models
• Fluent Alignment with Disfluent Judges: Post-training for Lower-resource Languages
• SPACE: Noise Contrastive Estimation Stabilizes Self-Play Fine-Tuning for Large Language Models
• Learning from Self Critique and Refinement for Faithful LLM Summarization
• AI & Human Co-Improvement for Safer Co-Superintelligence
• Self-Improving VLM Judges Without Human Annotations
• SR-GRPO: Stable Rank as an Intrinsic Geometric Reward for Large Language Model Alignment
• See, Think, Learn: A Self-Taught Multimodal Reasoner
• JarvisEvo: Towards a Self-Evolving Photo Editing Agent with Synergistic Editor-Evaluator Optimization
• Optimizing NetGPT via Routing-Based Synergy and Reinforcement Learning
• Graph-O1 : Monte Carlo Tree Search with Reinforcement Learning for Text-Attributed Graph Reasoning
• On Evaluating LLM Alignment by Evaluating LLMs as Judges
• ACT as Human: Multimodal Large Language Model Data Annotation with Critical Thinking
• From Exploration to Exploitation: A Two-Stage Entropy RLVR Approach for Noise-Tolerant MLLM Training
• You Need Reasoning to Learn Reasoning: The Limitations of Label-Free RL in Weak Base Models
• RLAC: Reinforcement Learning with Adversarial Critic for Free-Form Generation Tasks
• Autodata: An agentic data scientist to create high quality synthetic data
• Ouroboros-Spatial: Closing the Data-Model Loop for Spatial Reasoning
• LANPO: Bootstrapping Language and Numerical Feedback for Reinforcement Learning in LLMs
• SPICE: Self-Play In Corpus Environments Improves Reasoning
• Debiasing Reward Models by Representation Learning with Guarantees
• Multi-Agent Evolve: LLM Self-Improve through Co-evolution
• Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning
• Learning "Partner-Aware" Collaborators in Multi-Party Collaboration
• WebDevJudge: Evaluating (M)LLMs as Critiques for Web Development Quality
• Med-VRAgent: A Framework for Medical Visual Reasoning-Enhanced Agents
• Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domains
• Token-Level Inference-Time Alignment for Vision-Language Models
• Composition-Grounded Data Synthesis for Visual Reasoning
• Stop Reducing Responsibility in LLM-Powered Multi-Agent Systems to Local Alignment
• Training LLM Agents to Empower Humans
• Confidence as a Reward: Transforming LLMs into Reward Models
• Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanism
• Reinforced Preference Optimization for Recommendation
• Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey
• GTAlign: Game-Theoretic Alignment of LLM Assistants for Social Welfare
• Breaking the Self-Confirming Loop: Diagnosing and Mitigating Systemic Reward Bias in Self-Rewarding RL
• CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewards
• Self-Improving LLM Agents at Test-Time
• Aligning Large Language Models via Fully Self-Synthetic Data
• Reward Model Perspectives: Whose Opinions Do Reward Models Reward?
• RESTRAIN: From Spurious Votes to Signals -- Self-Driven RL with Self-Penalization
• Limited Preference Data? Learning Better Reward Model with Latent Space Synthesis
• Structural Reward Model: Enhancing Interpretability, Efficiency, and Scalability in Reward Modeling
• LatentEvolve: Self-Evolving Test-Time Scaling in Latent Space
• Mitigating Visual Hallucinations via Semantic Curriculum Preference Optimization in MLLMs
• Anchored Supervised Fine-Tuning
• SafeSearch: Automated Red-Teaming for the Safety of LLM-Based Search Agents
• Formalization Driven LLM Prompt Jailbreaking via Reinforcement Learning
• On the Shelf Life of Fine-Tuned LLM-Judges: Future-Proofing, Backward-Compatibility, and Question Generalization
• Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimization
• Self-Consistency as a Free Lunch: Reducing Hallucinations in Vision-Language Models via Self-Reflection
• DRIFT: Learning from Abundant User Dissatisfaction in Real-World Preference Learning
• Causally-Enhanced Reinforcement Policy Optimization
• Semantic Voting: A Self-Evaluation-Free Approach for Efficient LLM Self-Improvement on Unverifiable Open-ended Tasks
• RLP: Reinforcement as a Pretraining Objective
• TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them
• PromptCoT 2.0: Scaling Prompt Synthesis for Large Language Model Reasoning
• Rehearse: Stepping Back from the Confidence Cliff in Self-Improving Autoresearch
• References Improve LLM Alignment in Non-Verifiable Domains
• Enhancing Speech Large Language Models through Reinforced Behavior Alignment
• Can GRPO Boost Complex Multimodal Table Understanding?
• Self-Rewarding Rubric-Based Reinforcement Learning for Open-Ended Reasoning
• Language Self-Play For Data-Free Training
• Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Training
• Improving Large Vision and Language Models by Learning from a Panel of Peers
• Improving Alignment in LVLMs with Debiased Self-Judgment
• Self-Rewarding Vision-Language Model via Reasoning Decomposition
• Diversity First, Quality Later: A Two-Stage Assumption for Language Model Alignment
• Alleviating Attention Hacking in Discriminative Reward Modeling through Interaction Distillation
• Refine-n-Judge: Curating High-Quality Preference Chains for LLM-Fine-Tuning
• A Note on Code Quality Score: LLMs for Maintainable Large Codebases
• CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning tasks
• The Role of Feedback Alignment in Self-Distillation
• URPO: A Unified Reward & Policy Optimization Framework for Large Language Models
• Med-REFL: Medical Reasoning Enhancement via Self-Corrected Fine-grained Reflection
• Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning
• A Technical Survey of Reinforcement Learning Techniques for Large Language Models
• From Answers to Rationales: Self-Aligning Multimodal Reasoning with Answer-Oriented Chain-of-Thought
• SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning
• Bridging Offline and Online Reinforcement Learning for LLMs
• Enterprise Large Language Model Evaluation Benchmark
• From General to Targeted Rewards: Surpassing GPT-4 in Open-Ended Long-Context Generation
• Value-Free Policy Optimization via Reward Partitioning
• Unveiling the Learning Mind of Language Models: A Cognitive Framework and Empirical Study
• VL-GenRM: Enhancing Vision-Language Verification via Vision Experts and Iterative Training
• Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimization
• SPECS: Faster Test-Time Scaling through Speculative Drafts
• Bridging the Digital Divide: Small Language Models as a Pathway for Physics and Photonics Education in Underdeveloped Regions
• Feedback Friction: LLMs Struggle to Fully Incorporate External Feedback
• Training-free LLM Verification via Recycling Few-shot Examples
• PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifier
• LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learning
• Self-Improving Loops for Visual Robotic Planning
• Efficient Online RFT with Plug-and-Play LLM Judges: Unlocking State-of-the-Art Performance
• SPARTA ALIGNMENT: Collectively Aligning Multiple Language Models through Combat
• Protein Inverse Folding From Structure Feedback
• Understanding the Impact of Sampling Quality in Direct Preference Optimization
• Quantitative LLM Judges
• Iterative Self-Improvement of Vision Language Models for Image Scoring and Self-Explanation
• Self-Challenging Language Model Agents
• Reinforcing General Reasoning without Verifiers
• Learning to Reason without External Rewards
• SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning
• A Survey on Progress in LLM Alignment from the Perspective of Reward Design
• SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Data
• Reward Model Overoptimisation in Iterated RLHF
• Reward Reasoning Model
• Seek in the Dark: Reasoning via Test-Time Instance-Level Policy Gradient in Latent Space
• MR. Judge: Multimodal Reasoner as a Judge
• Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewards
• Synthetic Data RL: Task Definition Is All You Need
• T2A-Feedback: Improving Basic Capabilities of Text-to-Audio Generation via Fine-grained AI Feedback
• References Indeed Matter? Reference-Free Preference Optimization for Conversational Query Reformulation
• ComPO: Preference Alignment via Comparison Oracles
• DeepCritic: Deliberate Critique with Large Language Models
• R-C2: Cycle-Consistent Reinforcement Learning Improves Multimodal Reasoning
• Group-Evolving Agents: Open-Ended Self-Improvement via Experience Sharing
• Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models
• Self-CTRL: Self-Consistency Training with Reinforcement Learning
• GRPO Does Not Close the Multi-Agent Coordination Gap
• Toward Epistemic Stability: Engineering Consistent Procedures for Industrial LLM Hallucination Reduction
• ParamMem: Augmenting Language Agents with Parametric Reflective Memory
• Self-Improving Language Models with Bidirectional Evolutionary Search
• JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessment
• A New Strategy for Artificial Intelligence: Training Foundation Models Directly on Human Brain Data
• Learning from Language Feedback via Variational Policy Distillation
• Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Models
• Weak-to-Strong Generalization via Direct On-Policy Distillation
• Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents
• Contagion Networks: Evaluator Preference Propagation in Multi-Agent LLM Systems
• Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modeling
• Reinforcing Human Behavior Simulation via Verbal Feedback
• ARIS: Autonomous Research via Adversarial Multi-Agent Collaboration
• FinTradeBench: A Financial Reasoning Benchmark for LLMs
• Adaptive Helpfulness-Harmlessness Alignment with Preference Vectors
• Neural Garbage Collection: Learning to Forget while Learning to Reason
• Anyprefer: An Agentic Framework for Preference Data Synthesis
• Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision
• The Calibration Floor: Format Repair Can Masquerade as Self-Correction at Small-to-Mid Scale
• TTRL: Test-Time Reinforcement Learning
• Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
• EvalAgent: Discovering Implicit Evaluation Criteria from the Web
• Efficient MAP Estimation of LLM Judgment Performance with Prior Transfer
• Recursive Synthesis for Long-Horizon Terminal Tasks
• A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce
• SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skills
• EduPlanner: LLM-Based Multi-Agent Systems for Customized and Intelligent Instructional Design
• Recursive self-improvement [wikipedia]

Discussions

• Self-Rewarding Language Models [hn, 95 points, 60 comments]
• Self-Rewarding Language Models [hn, 3 points, 0 comments]
• Here's the link to Self-Rewarding Language Models, which shows that language models can criticize themselves well enough to improve themselves. arxiv.org/abs/2401.10020 #MLSky blueskAI [bsky, 2 points, 0 comments]
• Meta researchers introduce Self-Rewarding Language Models, aiming for superhuman agents. Their study shows these models, training via self-generated rewards, surpass existing systems like Claude 2 and [bsky, 1 points, 0 comments]

Related

• Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge
• Self-Evolved Reward Learning for LLMs
• Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Future
• Post-Training Large Language Models via Reinforcement Learning from Self-Feedback
• Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning