Prediction and Control in Continual Reinforcement Learning

2023/12/18 by Nishanth Anand, Doina Precup, Anand, Nishanth +1 · 3 citations
Computer Science · #Adaptive Dynamic Programming Control #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2312.11669

openalex publication_date 2023/12/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Temporal difference (TD) learning is often used to update the estimate of the value function which is used by RL agents to extract useful policies. In this paper, we focus on value function estimation in continual reinforcement learning. We propose to decompose the value function into two components which update at different timescales: a permanent value function, which holds general knowledge that persists over time, and a transient value function, which allows quick adaptation to new situations. We establish theoretical results showing that our approach is well suited for continual learning and draw connections to the complementary learning systems (CLS) theory from neuroscience. Empirically, this approach improves performance significantly on both prediction and control problems.

Cited by

• Towards better dense rewards in Reinforcement Learning Applications
• Continual Knowledge Adaptation for Reinforcement Learning
• Stackelberg Coupling of Online Representation Learning and Reinforcement Learning

Related

• Fixed-Horizon Temporal Difference Methods for Stable Reinforcement Learning
• Rethinking the Foundations for Continual Reinforcement Learning
• Recurrent Value Functions
• Continual Reinforcement Learning with Multi-Timescale Replay
• Towards Continual Reinforcement Learning: A Review and Perspectives