Reinforcement Learning of Markov Decision Processes with Peak Constraints

2019/01/23 by Ather Gattami, Gattami, Ather
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Optimization and Control (math.OC) #Optimization and Search Problems #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.1901.07839

openalex publication_date 2019/01/23 · openalex created_date 2019/02/21 · openalex updated_date 2026/07/28

Abstract

In this paper, we consider reinforcement learning of Markov Decision Processes (MDP) with peak constraints, where an agent chooses a policy to optimize an objective and at the same time satisfy additional constraints. The agent has to take actions based on the observed states, reward outputs, and constraint-outputs, without any knowledge about the dynamics, reward functions, and/or the knowledge of the constraint-functions. We introduce a game theoretic approach to construct reinforcement learning algorithms where the agent maximizes an unconstrained objective that depends on the simulated action of the minimizing opponent which acts on a finite set of actions and the output data of the constraint functions (rewards). We show that the policies obtained from maximin Q-learning converge to the optimal policies. To the best of our knowledge, this is the first time learning algorithms guarantee convergence to optimal stationary policies for the MDP problem with peak constraints for both discounted and expected average rewards.

Citations

• Constrained Markov Decision Processes

Related

• Reinforcement Learning for Multi-Objective and Constrained Markov Decision Processes
• Reinforcement Learning with Almost Sure Constraints
• Tackling Decision Processes with Non-Cumulative Objectives using Reinforcement Learning
• Reward is enough for convex MDPs
• Reinforcement Learning with Parameterized Actions