Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimism

2023/05/29 by Zihao Li, Zhuoran Yang, Li, Zihao +3 · 7 citations
Computer Science · Energy · Engineering · #Artificial Intelligence (cs.AI) #Energy, Environment, and Transportation Policies #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Optimization and Control (math.OC) #Reinforcement Learning in Robotics #Smart Grid Energy Management #Statistics Theory (math.ST)

paper · pdf · doi:10.48550/arxiv.2305.18438

openalex publication_date 2023/05/29 · openalex created_date 2023/06/03 · openalex updated_date 2026/07/28

Abstract

In this paper, we study offline Reinforcement Learning with Human Feedback (RLHF) where we aim to learn the human's underlying reward and the MDP's optimal policy from a set of trajectories induced by human choices. RLHF is challenging for multiple reasons: large state space but limited human feedback, the bounded rationality of human decisions, and the off-policy distribution shift. In this paper, we focus on the Dynamic Discrete Choice (DDC) model for modeling and understanding human choices. DCC, rooted in econometrics and decision theory, is widely used to model a human decision-making process with forward-looking and bounded rationality. We propose a \underlineDynamic-\underlineChoice-\underlinePessimistic-\underlinePolicy-\underlineOptimization (DCPPO) method. The method involves a three-stage process: The first step is to estimate the human behavior policy and the state-action value function via maximum likelihood estimation (MLE); the second step recovers the human reward function via minimizing Bellman mean squared error using the learned value functions; the third step is to plug in the learned reward and invoke pessimistic value iteration for finding a near-optimal policy. With only single-policy coverage (i.e., optimal policy) of the dataset, we prove that the suboptimality of DCPPO almost matches the classical pessimistic offline RL algorithm in terms of suboptimality's dependency on distribution shift and dimension. To the best of our knowledge, this paper presents the first theoretical guarantees for off-policy offline RLHF with dynamic discrete choice model.

Cited by

• From Passive to Persuasive: Localized Activation Injection for Empathy and Negotiation
• Offline Clustering of Preference Learning with Active-data Augmentation
• On the Role of Preference Variance in Preference Optimization
• LongRM: Revealing and Unlocking the Context Boundary of Reward Modeling
• Prompt and Parameter Co-Optimization for Large Language Models
• Understanding Post-Training Structural Changes in Large Language Models
• MultiFluxAI Enhancing Platform Engineering with Advanced Agent-Orchestrated Retrieval Systems

Related

• Reinforcement Learning from Human Feedback without Reward Inference: Model-Free Algorithm and Instance-Dependent Analysis
• A Minimaximalist Approach to Reinforcement Learning from Human Feedback
• Provable Offline Preference-Based Reinforcement Learning
• Strategyproof Reinforcement Learning from Human Feedback
• Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences