In value-based deep reinforcement learning, a pruned network is a good network

2024/02/19 by Johan Obando-Ceron, Aaron Courville, Obando-Ceron, Johan +3 · 1 voice · 7 citations
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural Networks and Reservoir Computing #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.2402.12479

openalex publication_date 2024/02/19 · arxiv published 2024/02/19 · openalex created_date 2024/02/22 · arxiv updated 2024/06/25 · openalex updated_date 2026/07/28

Abstract

Recent work has shown that deep reinforcement learning agents have difficulty in effectively using their network parameters. We leverage prior insights into the advantages of sparse training techniques and demonstrate that gradual magnitude pruning enables value-based agents to maximize parameter effectiveness. This results in networks that yield dramatic performance improvements over traditional networks, using only a small fraction of the full network parameters.

Cited by

• Representation Learning Enables Scalable Multitask Deep Reinforcement Learning
• Emergent Sparsity in Frozen Random CNN Feature Extractors for Deep Reinforcement Learning
• Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agents
• Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learning
• Geometric Properties of Neural Multivariate Regression
• Spectral Collapse Drives Loss of Plasticity in Deep Continual Learning
• Sparsity-Driven Plasticity in Multi-Task Reinforcement Learning

Discussions

• what about prior methods? let's start with pruning (from arxiv.org/abs/2402.12479 ) it turns out we don't have to prune the whole network, only the bottleneck! doing so yields even better performance [bsky, 3 points, 1 comments]

Related

• Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning
• Online Training and Pruning of Deep Reinforcement Learning Networks
• Learning to Prune Deep Neural Networks via Reinforcement Learning
• The State of Sparse Training in Deep Reinforcement Learning
• Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learning