Semi-supervised reward learning for offline reinforcement learning

2020/12/12 by Ksenia Konyushkova, Konyushkova, Ksenia, Konrad Zolna +12 · 4 citations
Computer Science · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification #Reinforcement Learning in Robotics #Robotics (cs.RO) #cs.AI #cs.LG #cs.RO

paper · pdf · doi:10.48550/arxiv.2012.06899

Accepted to Offline Reinforcement Learning Workshop at Neural Information Processing Systems (2020)

arxiv created 2020/12/12 · openalex publication_date 2020/12/12 · arxiv updated 2020/12/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In offline reinforcement learning (RL) agents are trained using a logged dataset. It appears to be the most natural route to attack real-life applications because in domains such as healthcare and robotics interactions with the environment are either expensive or unethical. Training agents usually requires reward functions, but unfortunately, rewards are seldom available in practice and their engineering is challenging and laborious. To overcome this, we investigate reward learning under the constraint of minimizing human reward annotations. We consider two types of supervision: timestep annotations and demonstrations. We propose semi-supervised learning algorithms that learn from limited annotations and incorporate unlabelled data. In our experiments with a simulated robotic arm, we greatly improve upon behavioural cloning and closely approach the performance achieved with ground truth rewards. We further investigate the relationship between the quality of the reward model and the final policies. We notice, for example, that the reward models do not need to be perfect to result in useful policies.

Citations

• End-to-End Robotic Reinforcement Learning without Reward Engineering
• BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Self-Training With Noisy Student Improves ImageNet Classification
• Offline Learning from Demonstrations and Unlabeled Experience

Cited by

• Offline Inverse Reinforcement Learning
• TROFI: Trajectory-Ranked Offline Inverse Reinforcement Learning
• Offline Meta-Reinforcement Learning with Online Self-Supervision
• Generalized Linear Markov Decision Process

Related

• How to Leverage Unlabeled Data in Offline Reinforcement Learning
• Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learning
• Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstration
• SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning
• Real-World Offline Reinforcement Learning from Vision Language Model Feedback