Q-function Decomposition with Intervention Semantics with Factored Action Spaces

2025/04/30 by Junkyu Lee, Tian Ming Gao, Lee, Junkyu +9
Computer Science · Mathematics · #Adaptive Dynamic Programming Control #Advanced Causal Inference Techniques #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2504.21326

openalex publication_date 2025/04/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Many practical reinforcement learning environments have a discrete factored action space that induces a large combinatorial set of actions, thereby posing significant challenges. Existing approaches leverage the regular structure of the action space and resort to a linear decomposition of Q-functions, which avoids enumerating all combinations of factored actions. In this paper, we consider Q-functions defined over a lower dimensional projected subspace of the original action space, and study the condition for the unbiasedness of decomposed Q-functions using causal effect estimation from the no unobserved confounder setting in causal statistics. This leads to a general scheme which we call action decomposed reinforcement learning that uses the projected Q-functions to approximate the Q-function in standard model-free reinforcement learning algorithms. The proposed approach is shown to improve sample complexity in a model-based reinforcement learning setting. We demonstrate improvements in sample efficiency compared to state-of-the-art baselines in online continuous control environments and a real-world offline sepsis treatment environment.

Citations

• Why Online Reinforcement Learning is Causal
• Counterfactual-Augmented Importance Sampling for Semi-Offline Policy Evaluation
• Leveraging Factored Action Spaces for Off-Policy Evaluation
• Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modeling
• Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare
• A Review of Off-Policy Evaluation in Reinforcement Learning
• Solving Continuous Control via Q-learning
• MoCoDA: Model-based Counterfactual Data Augmentation
• Causal Inference in Recommender Systems: A Survey and Future Directions
• Causal Inference in Recommender Systems: A Survey and Future Directions
• Weakly supervised causal representation learning
• Action-Sufficient State Representation Learning for Control with Structural Constraints
• Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings
• An Empirical Study of Representation Learning for Reinforcement Learning in Healthcare
• Learning to Represent Action Values as a Hypergraph on the Action Vertices
• Approximate information state for approximate planning and reinforcement learning in partially observed systems
• Representation Learning via Invariant Causal Mechanisms
• QPLEX: Duplex Dueling Multi-Agent Q-Learning
• Counterfactual Data Augmentation using Locally Factored Dynamics
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Causality for Machine Learning
• QTRAN: Learning to Factorize with Transformation for Cooperative\n Multi-Agent Reinforcement Learning
• Off-Policy Deep Reinforcement Learning without Exploration
• DeepMind Control Suite
• Learning Representations for Counterfactual Inference
• Deep Reinforcement Learning in Large Discrete Action Spaces
• Deep Reinforcement Learning with Double Q-learning
• Estimating causal effects of treatments in randomized and nonrandomized studies.
• Reinforcement Learning: An Introduction

Related

• Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare
• Quinoa: a Q-function You Infer Normalized Over Actions
• Stabilizing Q-learning with Linear Architectures for Provably Efficient Learning
• Discrete linear-complexity reinforcement learning in continuous action\n spaces for Q-learning algorithms
• Is Q-learning Provably Efficient?