When does Self-Prediction help? Understanding Auxiliary Tasks in Reinforcement Learning

2024/06/25 by Claas Voelcker, Tyler Kastner, Voelcker, Claas +5 · 5 citations
Psychology · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Mental Health Research Topics

paper · pdf · doi:10.48550/arxiv.2406.17718

openalex publication_date 2024/06/25 · openalex created_date 2024/06/27 · openalex updated_date 2026/07/28

Abstract

We investigate the impact of auxiliary learning tasks such as observation reconstruction and latent self-prediction on the representation learning problem in reinforcement learning. We also study how they interact with distractions and observation functions in the MDP. We provide a theoretical analysis of the learning dynamics of observation reconstruction, latent self-prediction, and TD learning in the presence of distractions and observation functions under linear model assumptions. With this formalization, we are able to explain why latent-self prediction is a helpful auxiliary task, while observation reconstruction can provide more useful features when used in isolation. Our empirical analysis shows that the insights obtained from our learning dynamics framework predicts the behavior of these loss functions beyond the linear model assumption in non-linear neural networks. This reinforces the usefulness of the linear model framework not only for theoretical analysis, but also practical benefit for applied problems.

Cited by

• Compositional Behavioral Semantics for State Abstraction in Reinforcement Learning
• TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learning
• Self-Predictive Representations for Combinatorial Generalization in Behavioral Cloning
• A Survey of State Representation Learning for Deep Reinforcement Learning
• Calibrated Value-Aware Model Learning with Probabilistic Environment Models
• Behavior-Consistent Deep Reinforcement Learning

Related

• Understanding Self-Predictive Learning for Reinforcement Learning
• Loss is its own Reward: Self-Supervision for Reinforcement Learning
• Predictive auxiliary objectives in deep RL mimic learning in the brain
• Bridging State and History Representations: Understanding Self-Predictive RL
• Data-Efficient Reinforcement Learning with Self-Predictive\n Representations