RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences

2024/02/27 by Jie Cheng, Gang Xiong, Cheng, Jie +9 · 12 citations
Decision Sciences · Engineering · #Multi-Criteria Decision Making #Smart Parking Systems Research

paper · pdf · doi:10.48550/arxiv.2402.17257

Abstract

Preference-based Reinforcement Learning (PbRL) circumvents the need for reward engineering by harnessing human preferences as the reward signal. However, current PbRL methods excessively depend on high-quality feedback from domain experts, which results in a lack of robustness. In this paper, we present RIME, a robust PbRL algorithm for effective reward learning from noisy preferences. Our method utilizes a sample selection-based discriminator to dynamically filter out noise and ensure robust training. To counteract the cumulative error stemming from incorrect selection, we suggest a warm start for the reward model, which additionally bridges the performance gap during the transition from pre-training to online training in PbRL. Our experiments on robotic manipulation and locomotion tasks demonstrate that RIME significantly enhances the robustness of the state-of-the-art PbRL method. Code is available at https://github.com/CJReinforce/RIME_ICML2024.

Cited by

• Explainable reinforcement learning from human feedback to improve alignment
• When Human Preferences Flip: An Instance-Dependent Robust Loss for RLHF
• Reinforcement Learning from Implicit Neural Feedback for Human-Aligned Robot Control
• Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignment
• STAIR: Addressing Stage Misalignment through Temporal-Aligned Preference Reinforcement Learning
• VRPO: Rethinking Value Modeling for Robust RL Training under Noisy Supervision
• Policy Learning from Large Vision-Language Model Feedback without Reward Modeling
• Aligning Humans and Robots via Reinforcement Learning from Implicit Human Feedback
• CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries
• DAPPER: Discriminability-Aware Policy-to-Policy Preference-Based Reinforcement Learning for Query-Efficient Robot Skill Acquisition
• TREND: Tri-teaching for Robust Preference-based Reinforcement Learning with Demonstrations
• Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning

Related

• Boosting Robustness in Preference-Based Reinforcement Learning with Dynamic Sparsity
• Efficient Preference-Based Reinforcement Learning Using Learned Dynamics Models
• Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewards
• STRAPPER: Preference-based Reinforcement Learning via Self-training Augmentation and Peer Regularization
• Residual Reward Models for Preference-based Reinforcement Learning