Counterfactual Data Augmentation using Locally Factored Dynamics

2020/07/06 by Silviu Pitis, Elliot Creager, Pitis, Silviu +3 · 20 citations
Computer Science · Mathematics · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics #Robotics (cs.RO) #cs.AI #cs.LG #cs.RO #stat.ML

paper · pdf · doi:10.48550/arxiv.2007.02863

In Proceedings of NeurIPS 2020. 10 pages (+5 references, +12 appendix). Code available at \url{https://github.com/spitis/mrl}

openalex publication_date 2020/07/06 · arxiv created 2020/12/03 · arxiv updated 2020/12/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Many dynamic processes, including common scenarios in robotic control and reinforcement learning (RL), involve a set of interacting subprocesses. Though the subprocesses are not independent, their interactions are often sparse, and the dynamics at any given time step can often be decomposed into locally independent causal mechanisms. Such local causal structures can be leveraged to improve the sample efficiency of sequence prediction and off-policy reinforcement learning. We formalize this by introducing local causal models (LCMs), which are induced from a global causal model by conditioning on a subset of the state space. We propose an approach to inferring these structures given an object-oriented state representation, as well as a novel algorithm for Counterfactual Data Augmentation (CoDA). CoDA uses local structures and an experience replay to generate counterfactual experiences that are causally valid in the global model. We find that CoDA significantly improves the performance of RL agents in locally factored tasks, including the batch-constrained and goal-conditioned settings.

Citations

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• Investigating Causal Relations by Econometric Models and Cross-spectral Methods
• Prioritized Experience Replay
• Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
• Certifying Some Distributional Robustness with Principled Adversarial Training
• The Effectiveness of Data Augmentation in Image Classification using Deep Learning
• Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines
• Addressing Function Approximation Error in Actor-Critic Methods
• Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
• Relational Deep Reinforcement Learning
• ARCHER: Aggressive Rewards to Counter bias in Hindsight Experience Replay
• Gaussian Error Linear Units (GELUs)
• Quantifying Generalization in Reinforcement Learning
• MONet: Unsupervised Scene Decomposition and Representation
• COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration
• Gradient-Based Neural DAG Learning
• Benchmarking Model-Based Reinforcement Learning
• Counterfactual Multi-Agent Policy Gradients
• Recurrent Independent Mechanisms
• Learning dexterous in-hand manipulation
• Contrastive Learning of Structured World Models
• Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning
• Causally Correct Partial Models for Reinforcement Learning
• Universal Value Density Estimation for Imitation Learning and Goal-Conditioned Reinforcement Learning
• Reinforcement Learning with Augmented Data
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems

Cited by

• Augmenting The Weather: A Hybrid Counterfactual-SMOTE Algorithm for Improving Crop Growth Prediction When Climate Changes
• Dynamic Sparsity: Challenging Common Sparsity Assumptions for Learning World Models in Robotic Reinforcement Learning Benchmarks
• Learning Interactive World Model for Object-Centric Reinforcement Learning
• CoRT: Counterfactual Replay for Token-Level Rubric-Guided Policy Optimization
• MoMaGen: Generating Demonstrations under Soft and Hard Constraints for Multi-Step Bimanual Mobile Manipulation
• DemoHLM: From One Demonstration to Generalizable Humanoid Loco-Manipulation
• From Pixels to Factors: Learning Independently Controllable State Variables for Reinforcement Learning
• CausalPlan: Empowering Efficient LLM Multi-Agent Collaboration Through Causality-Driven Planning
• Synthetic Data is Sufficient for Zero-Shot Visual Generalization from Offline Data
• Learning Causal Structure Distributions for Robust Planning
• Causal Process Models: Reframing Dynamic Causal Graph Discovery as a Reinforcement Learning Problem
• Hierarchical Reinforcement Learning with Targeted Causal Interventions
• Curious Causality-Seeking Agents Learn Meta Causal World
• Time Reversal Symmetry for Efficient Robotic Manipulations in Deep Reinforcement Learning
• On Measuring Intrinsic Causal Attributions in Deep Neural Networks
• Beyond the Known: Decision Making with Counterfactual Reasoning Decision Transformer
• Parameter Estimation using Reinforcement Learning Causal Curiosity: Limits and Challenges
• Learning Local Causal World Models with State Space Models and Attention
• Q-function Decomposition with Intervention Semantics with Factored Action Spaces
• Generative Framework for Personalized Persuasion: Inferring Causal, Counterfactual, and Latent Knowledge

Related

• MoCoDA: Model-based Counterfactual Data Augmentation
• Causal Action Influence Aware Counterfactual Data Augmentation
• Causal Counterfactuals for Improving the Robustness of Reinforcement Learning
• Fine-Grained Causal Dynamics Learning with Quantization for Improving Robustness in Reinforcement Learning
• Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentation