Learning Symbolic Persistent Macro-Actions for POMDP Solving Over Time

2025/05/06 by C Veronese, Veronese, Celeste, Daniele Meli +3
Computer Science · #Artificial Intelligence (cs.AI) #Computability, Logic, AI Algorithms #FOS: Computer and information sciences #Machine Learning and Algorithms #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2505.03668

openalex publication_date 2025/05/06 · openalex created_date 2025/10/16 · openalex updated_date 2026/07/28

Abstract

This paper proposes an integration of temporal logical reasoning and Partially Observable Markov Decision Processes (POMDPs) to achieve interpretable decision-making under uncertainty with macro-actions. Our method leverages a fragment of Linear Temporal Logic (LTL) based on Event Calculus (EC) to generate persistent (i.e., constant) macro-actions, which guide Monte Carlo Tree Search (MCTS)-based POMDP solvers over a time horizon, significantly reducing inference time while ensuring robust performance. Such macro-actions are learnt via Inductive Logic Programming (ILP) from a few traces of execution (belief-action pairs), thus eliminating the need for manually designed heuristics and requiring only the specification of the POMDP transition model. In the Pocman and Rocksample benchmark scenarios, our learned macro-actions demonstrate increased expressiveness and generality when compared to time-independent heuristics, indeed offering substantial computational efficiency improvements.

Citations

• Online inductive learning from answer sets for efficient reinforcement learning exploration
• Learning Logic Specifications for Soft Policy Guidance in POMCP
• Closing the Planning-Learning Loop with Application to Autonomous Driving
• Conflict-driven Inductive Logic Programming
• MAGIC: Learning Macro-Actions for Online POMDP Planning
• Approximate information state for approximate planning and reinforcement learning in partially observed systems
• Manipulation of Articulated Objects using Dual-arm Robots via Answer Set\n Programming
• Macro Action Reinforcement Learning with Sequence Disentanglement using Variational Autoencoder
• Planning and acting in partially observable stochastic domains

Related

• Counterexample-Guided Strategy Improvement for POMDPs Using Recurrent\n Neural Networks
• B³RTDP: A Belief Branch and Bound Real-Time Dynamic Programming Approach to Solving POMDPs
• Learning Logic Specifications for Soft Policy Guidance in POMCP
• MAGIC: Learning Macro-Actions for Online POMDP Planning
• Anytime Incremental ρPOMDP Planning in Continuous Spaces