Context-Sensitive Abstractions for Reinforcement Learning with Parameterized Actions

2025/12/23 by Rashmeet Kaur Nayyar, Nayyar, Rashmeet Kaur, Naman Shah +3
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #Robot Manipulation and Learning

paper · doi:10.48550/arxiv.2512.20831

openalex publication_date 2025/12/23 · openalex created_date 2025/12/26 · openalex updated_date 2026/07/28

Abstract

Real-world sequential decision-making often involves parameterized action spaces that require both, decisions regarding discrete actions and decisions about continuous action parameters governing how an action is executed. Existing approaches exhibit severe limitations in this setting -- planning methods demand hand-crafted action models, and standard reinforcement learning (RL) algorithms are designed for either discrete or continuous actions but not both, and the few RL methods that handle parameterized actions typically rely on domain-specific engineering and fail to exploit the latent structure of these spaces. This paper extends the scope of RL algorithms to long-horizon, sparse-reward settings with parameterized actions by enabling agents to autonomously learn both state and action abstractions online. We introduce algorithms that progressively refine these abstractions during learning, increasing fine-grained detail in the critical regions of the state-action space where greater resolution improves performance. Across several continuous-state, parameterized-action domains, our abstraction-driven approach enables TD(λ) to achieve markedly higher sample efficiency than state-of-the-art baselines.

Citations

• Expediting Reinforcement Learning by Incorporating Knowledge About Temporal Causality in the Environment
• Autonomous Option Invention for Continual Hierarchical Reinforcement Learning and Planning
• Continuous Control with Coarse-to-fine Reinforcement Learning
• Learning Abstract World Model for Value-preserving Planning with Options
• Building Minimal and Reusable Causal State Abstractions for Reinforcement Learning
• TD-MPC2: Scalable, Robust World Models for Continuous Control
• HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representation
• Polynomial Time Reinforcement Learning in Factored State MDPs with Linear Value Functions
• Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
• Multi-Pass Q-Networks for Deep Reinforcement Learning with Parameterised Action Spaces
• Hybrid Actor-Critic Reinforcement Learning in Parameterized Action Space
• Parametrized Deep Q-Networks Learning: Reinforcement Learning with Discrete-Continuous Hybrid Action Space
• Data-Efficient Hierarchical Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Proximal Policy Optimization Algorithms
• Deep Reinforcement Learning in Parameterized Action Space
• Continuous control with deep reinforcement learning
• Reinforcement Learning with Parameterized Actions

Related

• Discrete State-Action Abstraction via the Successor Representation
• Excluding the Irrelevant: Focusing Reinforcement Learning through Continuous Action Masking
• Autonomous Option Invention for Continual Hierarchical Reinforcement Learning and Planning
• Hierarchical Approaches for Reinforcement Learning in Parameterized Action Space
• Computationally Efficient Reinforcement Learning: Targeted Exploration leveraging Simple Rules