Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning

2017/03/06 by Joshua Achiam, S. Shankar Sastry, Achiam, Joshua +1 · 8 citations
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.1703.01732

openalex publication_date 2017/03/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Exploration in complex domains is a key challenge in reinforcement learning, especially for tasks with very sparse rewards. Recent successes in deep reinforcement learning have been achieved mostly using simple heuristic exploration strategies such as ε-greedy action selection or Gaussian control noise, but there are many tasks where these methods are insufficient to make any learning progress. Here, we consider more complex heuristics: efficient and scalable exploration strategies that maximize a notion of an agent's surprise about its experiences via intrinsic motivation. We propose to learn a model of the MDP transition probabilities concurrently with the policy, and to form intrinsic rewards that approximate the KL-divergence of the true transition probabilities from the learned model. One of our approximations results in using surprisal as intrinsic motivation, while the other gives the k-step learning progress. We show that our incentives enable agents to succeed in a wide range of environments with high-dimensional state spaces and very sparse rewards, including continuous control tasks and games in the Atari RAM domain, outperforming several other heuristic exploration techniques.

Citations

• Dueling Network Architectures for Deep Reinforcement Learning
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• Planning to Be Surprised: Optimal Bayesian Exploration in Dynamic Environments

Cited by

• The Information-Theoretic Imperative: Compression and the Epistemic Foundations of Intelligence
• Dense and Diverse Goal Coverage in Multi Goal Reinforcement Learning
• Leveraging Temporally Extended Behavior Sharing for Multi-task Reinforcement Learning
• LLM-Guided Task- and Affordance-Level Exploration in Reinforcement Learning
• Learning Gentle Object Manipulation with Curiosity-Driven Deep Reinforcement Learning
• Non-local Policy Optimization via Diversity-regularized Collaborative Exploration
• A Max-Min Entropy Framework for Reinforcement Learning
• Agent-centric learning: from external reward maximization to internal knowledge curation

Related

• Explore and Control with Adversarial Surprise
• Reinforcement Learning Generalization with Surprise Minimization
• Beyond Surprise: Improving Exploration Through Surprise Novelty
• VASE: Variational Assorted Surprise Exploration for Reinforcement Learning
• Successor-Predecessor Intrinsic Exploration