The Phenomenon of Policy Churn

2022/06/01 by Tom Schaul, André Barreto, André Sales Barreto +6 · 1 voice · 3 citations
Computer Science · #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2206.00730

Abstract

We identify and study the phenomenon of policy churn, that is, the rapid change of the greedy policy in value-based reinforcement learning. Policy churn operates at a surprisingly rapid pace, changing the greedy action in a large fraction of states within a handful of learning updates (in a typical deep RL set-up such as DQN on Atari). We characterise the phenomenon empirically, verifying that it is not limited to specific algorithm or environment properties. A number of ablations help whittle down the plausible explanations on why churn occurs to just a handful, all related to deep learning. Finally, we hypothesise that policy churn is a beneficial but overlooked form of implicit exploration that casts ε-greedy exploration in a fresh light, namely that ε-noise plays a much smaller role than expected.

Cited by

• Replicable Reinforcement Learning with Linear Function Approximation
• Forgetting is Everywhere
• ES-C51: Expected Sarsa Based C51 Distributional Reinforcement Learning Algorithm

Discussions

• Policy churn: maybe epsilon greedy doesn’t matter that much because the Q value argmax action changes constantly arxiv.org/abs/2206.00730 [bsky, 21 points, 2 comments]

Related

• Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn
• Understanding the Pathologies of Approximate Policy Evaluation when\n Combined with Greedification in Reinforcement Learning
• Convergence Guarantees for Deep Epsilon Greedy Policy Learning
• The Policy Cliff: A Theoretical Analysis of Reward-Policy Maps in Large Language Models
• Beyond the One Step Greedy Approach in Reinforcement Learning