Direct Preference-based Policy Optimization without Reward Modeling

2023/01/30 by Gaon An, An, Gaon, Junhyeok Lee +9 · 5 citations
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification #Reinforcement Learning in Robotics #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2301.12842

openalex publication_date 2023/01/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Preference-based reinforcement learning (PbRL) is an approach that enables RL agents to learn from preference, which is particularly useful when formulating a reward function is challenging. Existing PbRL methods generally involve a two-step procedure: they first learn a reward model based on given preference data and then employ off-the-shelf reinforcement learning algorithms using the learned reward model. However, obtaining an accurate reward model solely from preference information, especially when the preference is from human teachers, can be difficult. Instead, we propose a PbRL algorithm that directly learns from preference without requiring any reward modeling. To achieve this, we adopt a contrastive learning framework to design a novel policy scoring metric that assigns a high score to policies that align with the given preferences. We apply our algorithm to offline RL tasks with actual human preference labels and show that our algorithm outperforms or is on par with the existing PbRL methods. Notably, on high-dimensional control tasks, our algorithm surpasses offline RL methods that learn with ground-truth reward information. Finally, we show that our algorithm can be successfully applied to fine-tune large language models.

Cited by

• Safe and Optimal Learning from Preferences via Weighted Temporal Logic with Applications in Robotics and Formula 1
• Preference-Guided Learning for Sparse-Reward Multi-Agent Reinforcement Learning
• Implicit Safety Alignment from Crowd Preferences
• Policy Learning from Large Vision-Language Model Feedback without Reward Modeling
• TROFI: Trajectory-Ranked Offline Inverse Reinforcement Learning

Related

• Provable Reward-Agnostic Preference-Based Reinforcement Learning
• Policy-labeled Preference Learning: Is Preference Enough for RLHF?
• Beyond Reward: Offline Preference-guided Policy Optimization
• LEASE: Offline Preference-based Reinforcement Learning with High Sample Efficiency
• Contrastive Preference Learning: Learning from Human Feedback without RL