What Can Learned Intrinsic Rewards Capture?

2019/12/11 by Zeyu Zheng, Zheng, Zeyu, Junhyuk Oh +12 · 2 citations
Computer Science · Decision Sciences · Neuroscience · #Advanced Bandit Algorithms Research #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural dynamics and brain function #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.1912.05500

openalex publication_date 2019/12/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The objective of a reinforcement learning agent is to behave so as to maximise the sum of a suitable scalar function of state: the reward. These rewards are typically given and immutable. In this paper, we instead consider the proposition that the reward function itself can be a good locus of learned knowledge. To investigate this, we propose a scalable meta-gradient framework for learning useful intrinsic reward functions across multiple lifetimes of experience. Through several proof-of-concept experiments, we show that it is feasible to learn and capture knowledge about long-term exploration and exploitation into a reward function. Furthermore, we show that unlike policy transfer methods that capture "how" the agent should behave, the learned reward functions can generalise to other kinds of agents and to changes in the dynamics of the environment by capturing "what" the agent should strive to do.

Citations

• Humans use directed and random exploration to solve the explore–exploit dilemma.
• Learning to reinforcement learn
• RL²: Fast Reinforcement Learning via Slow Reinforcement Learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning
• Discovery of Useful Questions as Auxiliary Tasks
• Improving Generalization in Meta Reinforcement Learning using Learned Objectives

Cited by

• CORE: Compensable Reward as a Catalyst for Improving Offline RL in Wireless Networks
• Learning Task Belief Similarity with Latent Dynamics for Meta-Reinforcement Learning

Related

• Adapting Behaviour via Intrinsic Reward: A Survey and Empirical Study
• Pitfalls of learning a reward function online
• Meta-Learning to Explore via Memory Density Feedback
• On Learning Intrinsic Rewards for Policy Gradient Methods
• Generalizing Across Multi-Objective Reward Functions in Deep Reinforcement Learning