Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learning

2023/06/05 by Sam Lobel, Lobel, Sam, Akhil Bagaria +3 · 9 citations
Computer Science · #Artificial Intelligence (cs.AI) #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural Networks and Applications #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2306.03186

openalex publication_date 2023/06/05 · openalex created_date 2023/06/09 · openalex updated_date 2026/07/28

Abstract

We propose a new method for count-based exploration in high-dimensional state spaces. Unlike previous work which relies on density models, we show that counts can be derived by averaging samples from the Rademacher distribution (or coin flips). This insight is used to set up a simple supervised learning objective which, when optimized, yields a state's visitation count. We show that our method is significantly more effective at deducing ground-truth visitation counts than previous work; when used as an exploration bonus for a model-free reinforcement learning algorithm, it outperforms existing approaches on most of 9 challenging exploration tasks, including the Atari game Montezuma's Revenge.

Cited by

• EntroPIC: Towards Stable Long-Term Training of LLMs via Entropy Stabilization with Proportional-Integral Control
• Count Counts: Motivating Exploration in LLM Reasoning with Count-based Intrinsic Rewards
• Fill in the Blanks: Accelerating Q-Learning with a Handful of Demonstrations in Sparse Reward Settings
• Finite-time Convergence Analysis of Actor-Critic with Evolving Reward
• Analytical Survey of Learning with Low-Resource Data: From Analysis to Investigation
• Exploration by Random Reward Perturbation
• Uncertainty Prioritized Experience Replay
• Exploration Behavior of Untrained Policies
• Exploration by Random Distribution Distillation

Related

• Unifying Count-Based Exploration and Intrinsic Motivation
• Exploration in Feature Space for Reinforcement Learning
• Count-Based Exploration in Feature Space for Reinforcement Learning
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Approximate Exploration through State Abstraction