Information Directed Reward Learning for Reinforcement Learning

2021/02/24 by David Lindner, Matteo Turchetta, Lindner, David +7 · 5 citations
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Algorithms #Optimization and Search Problems

paper · pdf · doi:10.48550/arxiv.2102.12466

openalex publication_date 2021/02/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

For many reinforcement learning (RL) applications, specifying a reward is difficult. This paper considers an RL setting where the agent obtains information about the reward only by querying an expert that can, for example, evaluate individual states or provide binary preferences over trajectories. From such expensive feedback, we aim to learn a model of the reward that allows standard RL algorithms to achieve high expected returns with as few expert queries as possible. To this end, we propose Information Directed Reward Learning (IDRL), which uses a Bayesian model of the reward and selects queries that maximize the information gain about the difference in return between plausibly optimal policies. In contrast to prior active reward learning methods designed for specific types of queries, IDRL naturally accommodates different query types. Moreover, it achieves similar or better performance with significantly fewer queries by shifting the focus from reducing the reward approximation error to improving the policy induced by the reward model. We support our findings with extensive evaluations in multiple environments and with different query types.

Citations

• Scalable Bayesian Optimization Using Deep Neural Networks
• On a Measure of the Information Provided by an Experiment
• Scalable agent alignment via reward modeling: a research direction

Cited by

• Efficient Personalization of Generative Models via Optimal Experimental Design
• Optimistic Task Inference for Behavior Foundation Models
• Active Measuring in Reinforcement Learning With Delayed Negative Effects
• Which Rewards Matter? Reward Selection for Reinforcement Learning under Limited Feedback
• ASkDAgger: Active Skill-level Data Aggregation for Interactive Imitation Learning

Related

• Informativeness of Reward Functions in Reinforcement Learning
• Information Content Exploration
• Provable Reward-Agnostic Preference-Based Reinforcement Learning
• Robustness in the Face of Partial Identifiability in Reward Learning
• Information-Theoretic Reward Decomposition for Generalizable RLHF