Beyond the Known: Decision Making with Counterfactual Reasoning Decision Transformer

2025/05/14 by Minh Hoang Nguyen, Nguyen, Minh Hoang, Linh Le Pham Van +7 · 1 citation
Business, Management and Accounting · #Artificial Intelligence (cs.AI) #Big Data and Business Intelligence #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2505.09114

openalex publication_date 2025/05/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Decision Transformers (DT) play a crucial role in modern reinforcement learning, leveraging offline datasets to achieve impressive results across various domains. However, DT requires high-quality, comprehensive data to perform optimally. In real-world applications, the lack of training data and the scarcity of optimal behaviours make training on offline datasets challenging, as suboptimal data can hinder performance. To address this, we propose the Counterfactual Reasoning Decision Transformer (CRDT), a novel framework inspired by counterfactual reasoning. CRDT enhances DT ability to reason beyond known data by generating and utilizing counterfactual experiences, enabling improved decision-making in unseen scenarios. Experiments across Atari and D4RL benchmarks, including scenarios with limited data and altered dynamics, demonstrate that CRDT outperforms conventional DT approaches. Additionally, reasoning counterfactually allows the DT agent to obtain stitching abilities, combining suboptimal trajectories, without architectural modifications. These results highlight the potential of counterfactual reasoning to enhance reinforcement learning agents' performance and generalization capabilities.

Citations

• Reinformer: Max-Return Sequence Modeling for Offline RL
• Offline Trajectory Optimization for Offline Reinforcement Learning
• Policy Learning for Off-Dynamics RL with Deficient Support
• Offline Imitation Learning with Variational Counterfactual Reasoning
• Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Making
• Elastic Decision Transformer
• Future-conditioned Unsupervised Pretraining for Decision Transformer
• When should we prefer Decision Transformers for Offline Reinforcement Learning?
• MoCoDA: Model-based Counterfactual Data Augmentation
• Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL
• Estimating average causal effects from patient trajectories
• Online Decision Transformer
• Offline Reinforcement Learning with Implicit Q-Learning
• End-to-End Balancing for Causal Continuous Treatment-Effect Estimation
• Bellman-consistent Pessimism for Offline Reinforcement Learning
• Causal Influence Detection for Improving Efficiency in Reinforcement Learning
• Decision Transformer: Reinforcement Learning via Sequence Modeling
• Offline Reinforcement Learning with Fisher Divergence Critic Regularization
• Counterfactual Credit Assignment in Model-Free Reinforcement Learning
• Counterfactual Data Augmentation using Locally Factored Dynamics
• Counterfactually Guided Off-policy Transfer in Clinical Settings
• Conservative Q-Learning for Offline Reinforcement Learning
• MOPO: Model-based Offline Policy Optimization
• MOReL : Model-Based Offline Reinforcement Learning
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• G-Net: A Deep Learning Approach to G-computation for Counterfactual Outcome Prediction Under Dynamic Treatment Regimes
• Training Agents using Upside-Down Reinforcement Learning
• Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actions
• Supervised Reinforcement Learning with Recurrent Neural Network for\n Dynamic Treatment Recommendation
• Attention Is All You Need
• Reinforcement Learning: An Introduction

Cited by

• DmC: Nearest Neighbor Guidance Diffusion Model for Offline Cross-domain Reinforcement Learning

Related

• Transfer learning with causal counterfactual reasoning in Decision Transformers
• Predictive Coding for Decision Transformer
• DODT: Enhanced Online Decision Transformer Learning through Dreamer's Actor-Critic Trajectory Forecasting
• Rethinking Decision Transformer via Hierarchical Reinforcement Learning
• Learning impartial policies for sequential counterfactual explanations using Deep Reinforcement Learning