Preference Transformer: Modeling Human Preferences using Transformers for RL

2023/03/02 by Chang‐Yeon Kim, Kim, Changyeon, Jongjin Park +9 · 7 citations
Computer Science · #Anomaly Detection Techniques and Applications #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.2303.00957

openalex publication_date 2023/03/02 · openalex created_date 2023/03/05 · openalex updated_date 2026/07/28

Abstract

Preference-based reinforcement learning (RL) provides a framework to train agents using human preferences between two behaviors. However, preference-based RL has been challenging to scale since it requires a large amount of human feedback to learn a reward function aligned with human intent. In this paper, we present Preference Transformer, a neural architecture that models human preferences using transformers. Unlike prior approaches assuming human judgment is based on the Markovian rewards which contribute to the decision equally, we introduce a new preference model based on the weighted sum of non-Markovian rewards. We then design the proposed preference model using a transformer architecture that stacks causal and bidirectional self-attention layers. We demonstrate that Preference Transformer can solve a variety of control tasks using real human preferences, while prior approaches fail to work. We also show that Preference Transformer can induce a well-specified reward and attend to critical events in the trajectory by automatically capturing the temporal dependencies in human decision-making. Code is available on the project website: https://sites.google.com/view/preference-transformer.

Cited by

• LAPPI: Interactive Optimization with LLM-Assisted Preference-Based Problem Instantiation
• Reinforcement Learning from Implicit Neural Feedback for Human-Aligned Robot Control
• From Data to Rewards: a Bilevel Optimization Perspective on Maximum Likelihood Estimation
• STAIR: Addressing Stage Misalignment through Temporal-Aligned Preference Reinforcement Learning
• Preference-Guided Learning for Sparse-Reward Multi-Agent Reinforcement Learning
• Learning Real-World Acrobatic Flight from Human Preferences
• Policy Learning from Large Vision-Language Model Feedback without Reward Modeling

Related

• Contrastive Preference Learning: Learning from Human Feedback without RL
• Inverse Preference Learning: Preference-based RL without a Reward Function
• Reinforcement Learning from Diverse Human Preferences
• Policy-labeled Preference Learning: Is Preference Enough for RLHF?
• PrefMMT: Modeling Human Preferences in Preference-based Reinforcement Learning with Multimodal Transformers