Boosting Robustness in Preference-Based Reinforcement Learning with Dynamic Sparsity

2024/06/10 by Calarina Muslimani, Bram Grooten, Muslimani, Calarina +9 · 1 voice
Computer Science · Decision Sciences · #Multi-Criteria Decision Making #cs.LG

paper · pdf · doi:10.48550/arxiv.2406.06495

arxiv published 2024/06/10 · arxiv updated 2025/07/03

Abstract

To integrate into human-centered environments, autonomous agents must learn from and adapt to humans in their native settings. Preference-based reinforcement learning (PbRL) can enable this by learning reward functions from human preferences. However, humans live in a world full of diverse information, most of which is irrelevant to completing any particular task. It then becomes essential that agents learn to focus on the subset of task-relevant state features. To that end, this work proposes R2N (Robust-to-Noise), the first PbRL algorithm that leverages principles of dynamic sparse training to learn robust reward models that can focus on task-relevant features. In experiments with a simulated teacher, we demonstrate that R2N can adapt the sparse connectivity of its neural networks to focus on task-relevant features, enabling R2N to significantly outperform several sparse training and PbRL algorithms across simulated robotic environments.

Discussions

• We worked on making PbRL more robust to noisy envs recently: arxiv.org/abs/2406.06495 [bsky, 2 points, 1 comments]

Related

• RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences
• Efficient Preference-Based Reinforcement Learning Using Learned Dynamics Models
• Residual Reward Models for Preference-based Reinforcement Learning
• Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewards
• FLoRA: Sample-Efficient Preference-based RL via Low-Rank Style Adaptation of Reward Functions