Self-Composing Policies for Scalable Continual Reinforcement Learning

2025/06/04 by Mikel Malagón, Malagón, Mikel, Josu Ceberio +3 · 6 citations
Engineering · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Modular Robots and Swarm Intelligence

paper · pdf · doi:10.48550/arxiv.2506.14811

openalex publication_date 2025/06/04 · openalex created_date 2025/10/19 · openalex updated_date 2026/08/01

Abstract

This work introduces a growable and modular neural network architecture that naturally avoids catastrophic forgetting and interference in continual reinforcement learning. The structure of each module allows the selective combination of previous policies along with its internal policy, accelerating the learning process on the current task. Unlike previous growing neural network approaches, we show that the number of parameters of the proposed approach grows linearly with respect to the number of tasks, and does not sacrifice plasticity to scale. Experiments conducted in benchmark continuous control and visual problems reveal that the proposed approach achieves greater knowledge transfer and performance than alternative methods.

Citations

• DINOv2: Learning Robust Visual Features without Supervision
• Human-Timescale Adaptation in an Open-Ended Task Space
• Building a Subspace of Policies for Scalable Continual Learning
• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
• Disentangling Transfer in Continual Reinforcement Learning
• How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition
• Modular Lifelong Reinforcement Learning via Neural Composition
• Conflict-Averse Gradient Descent for Multi-task Learning
• Continual World: A Robotic Benchmark For Continual Reinforcement Learning
• Toward Robust Long Range Policy Transfer
• Towards Continual Reinforcement Learning: A Review and Perspectives
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Supermasks in Superposition
• Learning Invariant Representations for Reinforcement Learning without Reconstruction
• Agent57: Outperforming the Atari Human Benchmark
• Gradient Surgery for Multi-Task Learning
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Compacting, Picking and Growing for Unforgetting Continual Learning
• Routing Networks and the Challenges of Modular and Compositional Computation
• Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions
• On the role of neurogenesis in overcoming catastrophic forgetting
• Don't forget, there is more than forgetting: new metrics for Continual Learning
• Mix&Match - Agent Curricula for Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning
• Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning
• Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents
• Lifelong Learning with Dynamically Expandable Networks
• Proximal Policy Optimization Algorithms
• Attention Is All You Need
• Automated Curriculum Learning for Neural Networks
• Sim-to-Real Robot Learning from Pixels with Progressive Nets
• Progressive Neural Networks
• Policy Distillation
• Knowledge transfer in deep block-modular neural networks
• Playing Atari with Deep Reinforcement Learning
• Reinforcement Learning: An Introduction

Cited by

• Dynamic Mixture of Experts Against Severe Distribution Shifts
• Continual Knowledge Adaptation for Reinforcement Learning
• Policy Compatible Skill Incremental Learning via Lazy Learning Interface
• AblationBench: Evaluating Automated Planning of Ablations in Empirical AI Research
• A Survey of Continual Reinforcement Learning
• Curriculum-Adapted Robust Reinforcement Learning for UAV Deconfliction in Adversarial Environments

Related

• Reinforced Continual Learning
• Self-organization of action hierarchy and compositionality by reinforcement learning with recurrent neural networks
• Policy Consolidation for Continual Reinforcement Learning
• Continual Task Learning through Adaptive Policy Self-Composition
• Modular Continual Learning in a Unified Visual Environment