Equivalence of Optimality Criteria for Markov Decision Process and Model Predictive Control

2022/10/09 by Arash Bahari Kordabad, Kordabad, Arash Bahari, Mario Zanon +3
Computer Science · Engineering · #Advanced Control Systems Optimization #FOS: Electrical engineering #Formal Methods in Verification #Reinforcement Learning in Robotics #Systems and Control (eess.SY) #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2210.04302

openalex publication_date 2022/10/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper shows that the optimal policy and value functions of a Markov Decision Process (MDP), either discounted or not, can be captured by a finite-horizon undiscounted Optimal Control Problem (OCP), even if based on an inexact model. This can be achieved by selecting a proper stage cost and terminal cost for the OCP. A very useful particular case of OCP is a Model Predictive Control (MPC) scheme where a deterministic (possibly nonlinear) model is used to reduce the computational complexity. This observation leads us to parameterize an MPC scheme fully, including the cost function. In practice, Reinforcement Learning algorithms can then be used to tune the parameterized MPC scheme. We verify the developed theorems analytically in an LQR case and we investigate some other nonlinear examples in simulations.

Related

• Economic Model Predictive Control as a Solution to Markov Decision Processes
• Stability-Constrained Markov Decision Processes Using MPC
• Optimality Conditions for Model Predictive Control: Rethinking Predictive Model Design
• A General Markov Decision Process Framework for Directly Learning Optimal Control Policies
• Economic MPC of Markov Decision Processes: Dissipativity in Undiscounted Infinite-Horizon Optimal Control