1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities

2025/03/19 by Kevin Wang, Ishaan Javali, Wang, Kevin +7 · 4 voices · 12 citations
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.2503.14858

arxiv published 2025/03/19 · arxiv updated 2026/02/02

Abstract

Scaling up self-supervised learning has driven breakthroughs in language and vision, yet comparable progress has remained elusive in reinforcement learning (RL). In this paper, we study building blocks for self-supervised RL that unlock substantial improvements in scalability, with network depth serving as a critical factor. Whereas most RL papers in recent years have relied on shallow architectures (around 2 - 5 layers), we demonstrate that increasing the depth up to 1024 layers can significantly boost performance. Our experiments are conducted in an unsupervised goal-conditioned setting, where no demonstrations or rewards are provided, so an agent must explore (from scratch) and learn how to maximize the likelihood of reaching commanded goals. Evaluated on simulated locomotion and manipulation tasks, our approach increases performance on the self-supervised contrastive RL algorithm by 2× - 50×, outperforming other goal-conditioned baselines. Increasing the model depth not only increases success rates but also qualitatively changes the behaviors learned. The project webpage and code can be found here: https://wang-kevin3290.github.io/scaling-crl/.

Cited by

• Bridging Reinforcement Learning and Optimal Control via Feasible Action Mapping
• Do Depth-Grown Models Overcome the Curse of Depth? An In-Depth Analysis
• Is Temporal Difference Learning the Gold Standard for Stitching in RL?
• Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learning
• mHC-lite: You Don't Need 20 Sinkhorn-Knopp Iterations
• floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RL
• Compute-Optimal Scaling for Value-Based Deep RL
• Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategies
• The Serial Scaling Hypothesis
• Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning
• Flattening Hierarchies with Policy Bootstrapping
• On the Role of Computation in Reinforcement Learning

Discussions

• 1000 Layer Networks for Self-Supervised RL (NeurIPS 2025 Best Paper Award) [hn, 2 points, 0 comments]
• One Thousand Layer Networks for Self-Supervised RL [hn, 1 points, 0 comments]
• 9/ Thanks to amazing collaborators @IJ_Apps, @m_bortkiewicz @tomasztrzcinsk1, and @ben_eysenbach. Please check out our paper and website for more details! Paper: arxiv.org/abs/2503.14858 Website: wang [bsky, 0 points, 0 comments]
• 1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities arxiv.org/abs/2503.14858 Project page: wang-kevin3290.github.io/scaling-crl/ [bsky, 0 points, 0 comments]

Related

• Accelerating Goal-Conditioned RL Algorithms and Research
• Training Larger Networks for Deep Reinforcement Learning
• Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Data
• D2RL: Deep Dense Architectures in Reinforcement Learning
• Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategies