Continual State Representation Learning for Reinforcement Learning using Generative Replay

2018/10/09 by Hugo Caselles-Dupré, Michael Garcia-Ortiz, Caselles-Dupré, Hugo +4 · 19 citations
Computer Science · Mathematics · Physics and Astronomy · #Algorithm #Artificial intelligence #Bounded function #Computer science #Encoder #FOS: Computer and information sciences #Feature learning #Forgetting #Gaussian Processes and Bayesian Inference #Generative grammar #Generative model #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine learning #Mathematics #Model Reduction and Neural Networks #Reinforcement Learning in Robotics #Reinforcement learning #Representation (politics) #State (computer science) #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1810.03880

published in arXiv (Cornell University) (Cornell University) · Accepted contribution to the Workshop on Continual Learning, NeurIPS 2018 (Neural Information Processing Systems)

openalex publication_date 2018/10/09 · arxiv created 2018/12/11 · arxiv updated 2018/12/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We consider the problem of building a state representation model in a continual fashion. As the environment changes, the aim is to efficiently compress the sensory state's information without losing past knowledge. The learned features are then fed to a Reinforcement Learning algorithm to learn a policy. We propose to use Variational Auto-Encoders for state representation, and Generative Replay, i.e. the use of generated samples, to maintain past knowledge. We also provide a general and statistically sound method for automatic environment change detection. Our method provides efficient state representation as well as forward transfer, and avoids catastrophic forgetting. The resulting model is capable of incrementally learning information without using past data and with a bounded system size.

Citations

• Auto-Encoding Variational Bayes
• THE GENERALIZATION OF ‘STUDENT'S’ PROBLEM WHEN SEVERAL DIFFERENT POPULATION VARLANCES ARE INVOLVED
• Recurrent World Models Facilitate Policy Evolution
• Memory Replay GANs: learning to generate images from new categories without forgetting
• Flatland: a Lightweight First-Person 2-D Environment for Reinforcement Learning

Cited by

• Demonstration-Guided Continual Reinforcement Learning in Dynamic Environments
• GAN Memory with No Forgetting
• GRIm-RePR: Prioritising Generating Important Features for Pseudo-Rehearsal
• Fuzzy Tiling Activations: A Simple Approach to Learning Sparse Representations Online
• Dynamics-aware Embeddings
• S-TRIGGER: Continual State Representation Learning via Self-Triggered Generative Replay

Related

• S-TRIGGER: Continual State Representation Learning via Self-Triggered Generative Replay
• Continual Offline Reinforcement Learning via Diffusion-based Dual Generative Replay
• Continual Reinforcement Learning via Autoencoder-Driven Task and New Environment Recognition
• Continual Reinforcement Learning with Multi-Timescale Replay
• Continual Learning Using World Models for Pseudo-Rehearsal