Dynamical-VAE-based Hindsight to Learn the Causal Dynamics of Factored-POMDPs

2024/11/12 by Chao Han, Han, Chao, Debabrota Basu +7 · 1 citation
Computer Science · #Anomaly Detection Techniques and Applications #Bayesian Modeling and Causal Inference #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · pdf · doi:10.48550/arxiv.2411.07832

openalex publication_date 2024/11/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/02

Abstract

Learning representations of underlying environmental dynamics from partial observations is a critical challenge in machine learning. In the context of Partially Observable Markov Decision Processes (POMDPs), state representations are often inferred from the history of past observations and actions. We demonstrate that incorporating future information is essential to accurately capture causal dynamics and enhance state representations. To address this, we introduce a Dynamical Variational Auto-Encoder (DVAE) designed to learn causal Markovian dynamics from offline trajectories in a POMDP. Our method employs an extended hindsight framework that integrates past, current, and multi-step future information within a factored-POMDP setting. Empirical results reveal that this approach uncovers the causal graph governing hidden state transitions more effectively than history-based and typical hindsight-based models.

Cited by

• Learning Causal States Under Partial Observability and Perturbation

Related

• Learning Causal State Representations of Partially Observable Environments
• Counterfactual equivalence for POMDPs, and underlying deterministic\n environments
• Deep Belief Markov Models for POMDP Inference
• Flow-based Recurrent Belief State Learning for POMDPs
• Learning in POMDPs is Sample-Efficient with Hindsight Observability