The Dormant Neuron Phenomenon in Deep Reinforcement Learning

2023/02/24 by Ghada Sokar, Rishabh Agarwal, Sokar, Ghada +5 · 1 voice · 42 citations
Computer Science · Neuroscience · #Neural Networks and Applications #Neural dynamics and brain function #Reinforcement Learning in Robotics #cs.LG

paper · pdf · doi:10.48550/arxiv.2302.12902

arxiv published 2023/02/24 · arxiv updated 2023/06/13

Abstract

In this work we identify the dormant neuron phenomenon in deep reinforcement learning, where an agent's network suffers from an increasing number of inactive neurons, thereby affecting network expressivity. We demonstrate the presence of this phenomenon across a variety of algorithms and environments, and highlight its effect on learning. To address this issue, we propose a simple and effective method (ReDo) that Recycles Dormant neurons throughout training. Our experiments demonstrate that ReDo maintains the expressive power of networks by reducing the number of dormant neurons and results in improved performance.

Cited by

• Continual-RL for Generalization in Autonomous Racing on the RoboRacer Platform
• Calibrated Partial Resets: Preventing Policy Collapse in Continual Reinforcement Learning
• Bridging Streaming Continual Learning via In-Context Large Tabular Models
• Iterated Population Based Training with Task-Agnostic Restarts
• Representation Learning Enables Scalable Multitask Deep Reinforcement Learning
• Emergent Sparsity in Frozen Random CNN Feature Extractors for Deep Reinforcement Learning
• The Three Regimes of Offline-to-Online Reinforcement Learning
• Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agents
• Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learning
• Pruning Cannot Hurt Robustness: Certified Trade-offs in Reinforcement Learning
• Egalitarian Gradient Descent: A Simple Approach to Accelerated Grokking
• Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoning
• Barriers for Learning in an Evolving World: Mathematical Understanding of Loss of Plasticity
• Activation Function Design Sustains Plasticity in Continual Learning
• Spectral Collapse Drives Loss of Plasticity in Deep Continual Learning
• The Kinetics of Training: A Driven-Nucleation Rate Law for Emergence, Plasticity Loss, and Circuit Control in Language Models
• One Model for All Tasks: Leveraging Efficient World Models in Multi-Task Planning
• HERMES: Human-to-Robot Embodied Learning from Multi-Source Motion Data for Mobile Dexterous Manipulation
• Sparsity-Driven Plasticity in Multi-Task Reinforcement Learning
• Sample-efficient LLM Optimization with Reset Replay
• Reinitializing weights vs units for maintaining plasticity in neural networks
• Balancing Expressivity and Robustness: Constrained Rational Activations for Reinforcement Learning
• A Simple Baseline for Stable and Plastic Neural Networks
• Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Data
• Recovering Plasticity of Neural Networks via Soft Weight Rescaling
• A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
• Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning
• Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning
• MEAL: A Benchmark for Continual Multi-Agent Reinforcement Learning
• The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning
• The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks
• Optimistic critics can empower small actors
• Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn
• Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learning
• The Cell Must Go On: Agar.io for Continual Reinforcement Learning
• Hadamax Encoding: Elevating Performance in Model-Free Atari
• Preserving Plasticity in Continual Learning with Adaptive Linearity Injection
• Understanding and Exploiting Plasticity for Non-stationary Network Resource Adaptation
• Stable Deep Reinforcement Learning via Isotropic Gaussian Representations
• Plasticine: Accelerating Research in Plasticity-Motivated Deep Reinforcement Learning
• Evaluating Temporal Plasticity in Foundation Time Series Models for Incremental Fine-tuning
• Plasticity-Aware Mixture of Experts for Learning Under QoE Shifts in Adaptive Video Streaming

Discussions

• Is it all related to dormant neurons or is there other literature on why RL struggles with plasticity? arxiv.org/abs/2302.12902 [bsky, 2 points, 1 comments]

Related

• Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learning
• PRIME: Plasticity Recovery in Multi-Agent Environments for UAV-Assisted Emergency Communication Networks
• Neuron-level Balance between Stability and Plasticity in Deep Reinforcement Learning
• AltNet: Addressing the Plasticity-Stability Dilemma in Reinforcement Learning
• Continual Learning with Deep Artificial Neurons