On Pioneering Works of Albert Shiryaev on Markov Decision Processes and Some Later Developments

2025/06/05 by Eugene A. Feinberg, Feinberg, Eugene A.
Computer Science · Engineering · #FOS: Mathematics #Optimization and Control (math.OC) #Probability (math.PR) #Reinforcement Learning in Robotics #Stability and Control of Uncertain Systems #Target Tracking and Data Fusion in Sensor Networks

paper · pdf · doi:10.48550/arxiv.2506.04896

openalex publication_date 2025/06/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This article is dedicated to three fundamental papers on Markov Decision Processes and on control with incomplete observations published by Albert Shiryaev approximately sixty years ago. One of these papers was coauthored with O.V. Viskov. We discuss some of the results and some of many rich ideas presented in these papers and survey some later developments. At the end we mention some recent studies of Albert Shiryaev on Kolmogorov's equations for jump Markov processes and on control of continuous-time jump Markov processes.

Citations

• Kolmogorov's Equations for Jump Markov Processes and their Applications\n to Control Problems
• Markov Decision Processes with Incomplete Information and Semi-Uniform Feller Transition Probabilities
• MDPs with Setwise Continuous Transition Probabilities
• Weak Feller Property of Non-linear Filters
• Kolmogorov's Equations for Jump Markov Processes with Unbounded Jump\n Rates
• Partially Observable Total-Cost Markov Decision Processes with Weakly Continuous Transition Probabilities
• The Value Iteration Algorithm is Not Strongly Polynomial for Discounted Dynamic Programming
• Examples Concerning Abelian and Cesaro Limits
• Berge's Maximum Theorem for Noncompact Image Sets
• Bergeʼs maximum theorem for noncompact image sets
• Improved and Generalized Upper Bounds on the Complexity of Policy Iteration
• On solutions of Kolmogorov's equations for jump Markov processes
• The simplex method is strongly polynomial for deterministic Markov decision processes
• Berge's Theorem for Noncompact Image Sets
• Average-Cost Markov Decision Processes with Weakly Continuous Transition\n Probabilities
• A Bound for the Number of Different Basic Solutions Generated by the Simplex Method
• Reduction of a Controlled Markov Model with Incomplete Data to a Problem with Complete Information in the Case of Borel State and Control Space
• Multichain Markov Renewal Programs

Related

• Kolmogorov's Equations for Jump Markov Processes and their Applications\n to Control Problems
• On solutions of Kolmogorov's equations for jump Markov processes
• Kolmogorov's Equations for Jump Markov Processes with Unbounded Jump\n Rates
• Lecture Notes on Stochastic Processes
• Cascade Markov Decision Processes: Theory and Applications