Convergence and stability of Q-learning in Hierarchical Reinforcement Learning

2025/11/21 by Massimiliano Manenti, Manenti, Massimiliano, Andrea Iannelli +1
Computer Science · #Reinforcement Learning in Robotics #Adaptive Dynamic Programming Control #Domain Adaptation and Few-Shot Learning

paper · pdf · doi:10.48550/arxiv.2511.17351

Abstract

Hierarchical Reinforcement Learning promises, among other benefits, to efficiently capture and utilize the temporal structure of a decision-making problem and to enhance continual learning capabilities, but theoretical guarantees lag behind practice. In this paper, we propose a Feudal Q-learning scheme and investigate under which conditions its coupled updates converge and are stable. By leveraging the theory of Stochastic Approximation and the ODE method, we present a theorem stating the convergence and stability properties of Feudal Q-learning. This provides a principled convergence and stability analysis tailored to Feudal RL. Moreover, we show that the updates converge to a point that can be interpreted as an equilibrium of a suitably defined game, opening the door to game-theoretic approaches to Hierarchical RL. Lastly, experiments based on the Feudal Q-learning algorithm support the outcomes anticipated by theory.

Citations

• O(1/k) Finite-Time Bound for Non-Linear Two-Time-Scale Stochastic Approximation
• Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks
• Safe Hierarchical Model Predictive Control and Planning for Autonomous Systems
• Multi Timescale Stochastic Approximation: Stability and Convergence
• Nonlinear Two-Time-Scale Stochastic Approximation: Convergence and Finite-Time Performance
• Game-Theoretic Multiagent Reinforcement Learning
• Diversity-enabled sweet spots in layered architectures and\n speed-accuracy trade-offs in sensorimotor control
• DAC: The Double Actor-Critic Architecture for Learning Options
• Learning Abstract Options
• Is Q-learning Provably Efficient?
• Data-Efficient Hierarchical Reinforcement Learning
• On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learning
• FeUdal Networks for Hierarchical Reinforcement Learning
• The Option-Critic Architecture
• Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
• Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition
• A Stochastic Approximation Method

Related

• Reinforcement learning in convergently non-stationary environments: Feudal hierarchies and learned representations
• Asymptotic Convergence and Performance of Multi-Agent Q-Learning Dynamics
• Stabilizing Q-learning with Linear Architectures for Provably Efficient Learning
• Q-learning as a monotone scheme
• An Elementary Proof that Q-learning Converges Almost Surely