2024/03/15 by Guoxi Zhang, Han Bao, Zhang, Guoxi +3
Decision Sciences · Economics, Econometrics and Finance · #Auction Theory and Applications #Economic Policies and Impacts #FOS: Computer and information sciences #Game Theory and Voting Systems #Machine Learning (cs.LG)
paper · pdf · doi:10.48550/arxiv.2403.10160
openalex publication_date 2024/03/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
In preference-based reinforcement learning (PbRL), a reward function is learned from a type of human feedback called preference. To expedite preference collection, recent works have leveraged offline preferences, which are preferences collected for some offline data. In this scenario, the learned reward function is fitted on the offline data. If a learning agent exhibits behaviors that do not overlap with the offline data, the learned reward function may encounter generalizability issues. To address this problem, the present study introduces a framework that consolidates offline preferences and virtual preferences for PbRL, which are comparisons between the agent's behaviors and the offline data. Critically, the reward function can track the agent's behaviors using the virtual preferences, thereby offering well-aligned guidance to the agent. Through experiments on continuous control tasks, this study demonstrates the effectiveness of incorporating the virtual preferences in PbRL.