Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search

2018/11/15 by Lars Buesing, Théophane Weber, Buesing, Lars +11 · 11 citations
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.1811.06272

openalex publication_date 2018/11/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Learning policies on data synthesized by models can in principle quench the thirst of reinforcement learning algorithms for large amounts of real experience, which is often costly to acquire. However, simulating plausible experience de novo is a hard problem for many complex environments, often resulting in biases for model-based policy evaluation and search. Instead of de novo synthesis of data, here we assume logged, real experience and model alternative outcomes of this experience under counterfactual actions, actions that were not actually taken. Based on this, we propose the Counterfactually-Guided Policy Search (CF-GPS) algorithm for learning policies in POMDPs from off-policy experience. It leverages structural causal models for counterfactual evaluation of arbitrary policies on individual off-policy episodes. CF-GPS can improve on vanilla model-based RL algorithms by making use of available logged data to de-bias model predictions. In contrast to off-policy algorithms based on Importance Sampling which re-weight data, CF-GPS leverages a model to explicitly consider alternative outcomes, allowing the algorithm to make better use of experience data. We find empirically that these advantages translate into improved policy evaluation and search results on a non-trivial grid-world task. Finally, we show that CF-GPS generalizes the previously proposed Guided Policy Search and that reparameterization-based algorithms such Stochastic Value Gradient can be interpreted as counterfactual methods.

Citations

• Adam: A Method for Stochastic Optimization
• DRAW: A Recurrent Neural Network For Image Generation
• Stochastic Backpropagation and Approximate Inference in Deep Generative\n Models
• A comparative study of counterfactual estimators
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures

Cited by

• Unifying Causal Reinforcement Learning: Survey, Taxonomy, Algorithms and Applications
• CauSight: Learning to Supersense for Visual Causal Discovery
• Synthetic Returns for Long-Term Credit Assignment
• ExoPredicator: Learning Abstract Models of Dynamic Worlds for Robot Planning
• Goal Discovery with Causal Capacity for Efficient Reinforcement Learning
• Causal Process Models: Reframing Dynamic Causal Graph Discovery as a Reinforcement Learning Problem
• Text-Driven Causal Representation Learning for Source-Free Domain Generalization
• A New Perspective On AI Safety Through Control Theory Methodologies
• Causal-Paced Deep Reinforcement Learning
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• Causality in the human niche: lessons for machine learning

Related

• Learning Efficient and Effective Exploration Policies with Counterfactual Meta Policy
• Random Actions vs Random Policies: Bootstrapping Model-Based Direct Policy Search
• Adversarial Counterfactual Environment Model Learning
• When to Trust Your Model: Model-Based Policy Optimization
• Towards Causal Model-Based Policy Optimization