Transfer learning with causal counterfactual reasoning in Decision Transformers

2021/10/27 by Ayman Boustati, Boustati, Ayman, Hana Chockler +3
Computer Science · Energy · #Artificial Intelligence (cs.AI) #Data Stream Mining Techniques #Energy Efficiency and Management #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2110.14355

openalex publication_date 2021/10/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The ability to adapt to changes in environmental contingencies is an important challenge in reinforcement learning. Indeed, transferring previously acquired knowledge to environments with unseen structural properties can greatly enhance the flexibility and efficiency by which novel optimal policies may be constructed. In this work, we study the problem of transfer learning under changes in the environment dynamics. In this study, we apply causal reasoning in the offline reinforcement learning setting to transfer a learned policy to new environments. Specifically, we use the Decision Transformer (DT) architecture to distill a new policy on the new environment. The DT is trained on data collected by performing policy rollouts on factual and counterfactual simulations from the source environment. We show that this mechanism can bootstrap a successful policy on the target environment while retaining most of the reward.

Related

• Beyond the Known: Decision Making with Counterfactual Reasoning Decision Transformer
• Towards Causal Model-Based Policy Optimization
• Transfer Reinforcement Learning under Unobserved Contextual Information
• Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learning
• Dichotomy of Control: Separating What You Can Control from What You Cannot