Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards

2025/05/05 by Xiaobao Wu, Wu, Xiaobao · 6 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning and Algorithms #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2505.02686

openalex publication_date 2025/05/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent developments in Large Language Models (LLMs) have shifted from pre-training scaling to post-training and test-time scaling. Across these developments, a key unified paradigm has arisen: Learning from Rewards, where reward signals act as the guiding stars to steer LLM behavior. It has underpinned a wide range of prevalent techniques, such as reinforcement learning (RLHF, RLAIF, DPO, and GRPO), reward-guided decoding, and post-hoc correction. Crucially, this paradigm enables the transition from passive learning from static data to active learning from dynamic feedback. This endows LLMs with aligned preferences and deep reasoning capabilities for diverse tasks. In this survey, we present a comprehensive overview of learning from rewards, from the perspective of reward models and learning strategies across training, inference, and post-inference stages. We further discuss the benchmarks for reward models and the primary applications. Finally we highlight the challenges and future directions. We maintain a paper collection at https://github.com/bobxwu/learning-from-rewards-llm-papers.

Cited by

• AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agent
• Multidimensional Rubric-oriented Reward Model Learning via Geometric Projection Reference Constraints
• Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domains
• A Survey of Inductive Reasoning for Large Language Models
• Unsupervised Hallucination Detection by Inspecting Reasoning Processes
• Graph-R1: Towards Agentic GraphRAG Framework via End-to-end Reinforcement Learning

Related

• Generalist Reward Models: Found Inside Large Language Models
• Cooper: Co-Optimizing Policy and Reward Models in Reinforcement Learning for Large Language Models
• Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey
• R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learning
• RewardBench: Evaluating Reward Models for Language Modeling