Learn to Change the World: Multi-level Reinforcement Learning with Model-Changing Actions

2025/10/16 by Ziqing Lu, Lu, Ziqing, Babak Hassibi +5
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2510.15056

openalex publication_date 2025/10/16 · openalex created_date 2025/10/21 · openalex updated_date 2026/07/28

Abstract

Reinforcement learning usually assumes a given or sometimes even fixed environment in which an agent seeks an optimal policy to maximize its long-term discounted reward. In contrast, we consider agents that are not limited to passive adaptations: they instead have model-changing actions that actively modify the RL model of world dynamics itself. Reconfiguring the underlying transition processes can potentially increase the agents' rewards. Motivated by this setting, we introduce the multi-layer configurable time-varying Markov decision process (MCTVMDP). In an MCTVMDP, the lower-level MDP has a non-stationary transition function that is configurable through upper-level model-changing actions. The agent's objective consists of two parts: Optimize the configuration policies in the upper-level MDP and optimize the primitive action policies in the lower-level MDP to jointly improve its expected long-term reward.

Citations

• RL²: Fast Reinforcement Learning via Slow Reinforcement Learning
• Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning

Related

• Non-stationary and Varying-discounting Markov Decision Processes for Reinforcement Learning
• Reinforcement Learning with an Abrupt Model Change
• How RL Agents Behave When Their Actions Are Modified
• Adaptive Multi-Horizon Reinforcement Learning
• Intrinsically Motivated Hierarchical Policy Learning in Multi-objective Markov Decision Processes