A Survey on Progress in LLM Alignment from the Perspective of Reward Design

2025/05/05 by Min Ji, Yonghui Wu, Ji, Miaomiao +11 · 7 citations
Environmental Science · #Educational Reforms and Innovations

paper · pdf · doi:10.48550/arxiv.2505.02666

Abstract

Reward design plays a pivotal role in aligning large language models (LLMs) with human values, serving as the bridge between feedback signals and model optimization. This survey provides a structured organization of reward modeling and addresses three key aspects: mathematical formulation, construction practices, and interaction with optimization paradigms. Building on this, it develops a macro-level taxonomy that characterizes reward mechanisms along complementary dimensions, thereby offering both conceptual clarity and practical guidance for alignment research. The progression of LLM alignment can be understood as a continuous refinement of reward design strategies, with recent developments highlighting paradigm shifts from reinforcement learning (RL)-based to RL-free optimization and from single-task to multi-objective and complex settings.

Cited by

• Multidimensional Rubric-oriented Reward Model Learning via Geometric Projection Reference Constraints
• On the Role of Preference Variance in Preference Optimization
• GTAlign: Game-Theoretic Alignment of LLM Assistants for Social Welfare
• Unlocking the Essence of Beauty: Advanced Aesthetic Reasoning with Relative-Absolute Policy Optimization
• We Think, Therefore We Align LLMs to Helpful, Harmless and Honest Before They Go Wrong
• Pluralistic Off-policy Evaluation and Alignment
• Eliciting and Analyzing Emergent Misalignment in State-of-the-Art Large Language Models

Related

• Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment
• RMB: Comprehensively Benchmarking Reward Models in LLM Alignment
• Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
• Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment
• Beyond Monolithic Rewards: A Hybrid and Multi-Aspect Reward Optimization for MLLM Alignment