Causal Reasoning from Meta-reinforcement Learning

2019/01/23 by Ishita Dasgupta, Jane Wang, Jane X. Wang +20 · 1 voice · 18 citations
Computer Science · Mathematics · #AI-based Problem Solving and Planning #Bayesian Modeling and Causal Inference #Reinforcement Learning in Robotics #cs.AI #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1901.08162

arxiv created 2019/01/23 · openalex publication_date 2019/01/23 · arxiv updated 2019/01/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Discovering and exploiting the causal structure in the environment is a crucial challenge for intelligent agents. Here we explore whether causal reasoning can emerge via meta-reinforcement learning. We train a recurrent network with model-free reinforcement learning to solve a range of problems that each contain causal structure. We find that the trained agent can perform causal reasoning in novel situations in order to obtain rewards. The agent can select informative interventions, draw causal inferences from observational data, and make counterfactual predictions. Although established formal causal reasoning algorithms also exist, in this paper we show that such reasoning can arise from model-free reinforcement learning, and suggest that causal reasoning in complex settings may benefit from the more end-to-end learning-based approaches presented here. This work also offers new strategies for structured exploration in reinforcement learning, by providing agents with the ability to perform -- and interpret -- experiments.

Cited by

• Unifying Causal Reinforcement Learning: Survey, Taxonomy, Algorithms and Applications
• CauSight: Learning to Supersense for Visual Causal Discovery
• DODO: Causal Structure Learning with Budgeted Interventions
• A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms
• Meta-Learning of Structured Task Distributions in Humans and Machines
• Anatomy of a Machine Learning Ecosystem: 2 Million Models on Hugging Face
• Causal Induction from Visual Observations for Goal Directed Tasks
• Adversarial Examples and the Deeper Riddle of Induction: The Need for a Theory of Artifacts in Deep Learning
• A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions
• A Comprehensive Overview and Survey of Recent Advances in Meta-Learning
• Bhatt Conjectures: On Necessary-But-Not-Sufficient Benchmark Tautology for Human Like Reasoning
• Efficiently Disentangle Causal Representations
• BADGER: Learning to (Learn [Learning Algorithms] through Multi-Agent Communication)
• Towards intervention-centric causal reasoning in learning agents
• Where Do Human Heuristics Come From?
• Interventional Sum-Product Networks: Causal Inference with Tractable\n Probabilistic Models
• A Theoretical Analysis of Compositional Generalization in Neural Networks: A Necessary and Sufficient Condition
• Parameter Estimation using Reinforcement Learning Causal Curiosity: Limits and Challenges

Discussions

• Causal Reasoning from Meta-Reinforcement Learning [hn, 7 points, 0 comments]

Related

• A Meta-Reinforcement Learning Algorithm for Causal Discovery
• Towards intervention-centric causal reasoning in learning agents
• Unifying Causal Reinforcement Learning: Survey, Taxonomy, Algorithms and Applications
• Causal Reinforcement Learning: A Survey
• Why Online Reinforcement Learning is Causal