Yet Another Distributional Bellman Equation

2025/05/27 by Nicole Bäuerle, Bäuerle, Nicole, Tamara Göll +3
Economics, Econometrics and Finance · Mathematics · Computer Science · #Stochastic processes and financial applications #Mathematical and Theoretical Analysis #Chaos, Complexity, and Education

paper · pdf · doi:10.48550/arxiv.2505.21098

Abstract

We consider non-standard Markov Decision Processes (MDPs) where the target function is not only a simple expectation of the accumulated reward. Instead, we consider rather general functionals of the joint distribution of terminal state and accumulated reward which have to be optimized. For finite state and compact action space, we show how to solve these problems by defining a lifted MDP whose state space is the space of distributions over the true states of the process. We derive a Bellman equation in this setting, which can be considered as a distributional Bellman equation. Well-known cases like the standard MDP and quantile MDPs are shown to be special examples of our framework. We also apply our model to a variant of an optimal transport problem.

Citations

• Optimizing Return Distributions with Distributional Dynamic Programming
• Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional Reinforcement Learning
• Markov Decision Processes with Risk-Sensitive Criteria: An Overview
• Beyond Average Return in Markov Decision Processes
• Time-consistency in the mean-variance problem: A new perspective
• On solutions of the distributional Bellman equation
• Minimizing Spectral Risk Measures Applied to Markov Decision Processes
• Bicausal Optimal Transport for Markov Chains via Dynamic Programming
• Personalized Robo-Advising: Enhancing Investment through Client\n Interaction
• Statistics and Samples in Distributional Reinforcement Learning
• A Comparative Analysis of Expected and Distributional Reinforcement\n Learning
• Implicit Quantile Networks for Distributional Reinforcement Learning
• Distributional Reinforcement Learning with Quantile Regression
• A Distributional Perspective on Reinforcement Learning
• Optimizing Quantiles in Preference-based Markov Decision Processes
• Causal transport in discrete time and applications
• Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach
• Unified Framework of Mean-Field Formulations for Optimal Multi-period Mean-Variance Portfolio Selection
• Mean-Variance Optimization in Markov Decision Processes
• Utility theory front to back - inferring utility from agents' choices
• Constrained Markov Decision Processes

Related

• A Counterexample and a Corrective to the Vector Extension of the Bellman Equations of a Markov Decision Process
• Mean field for Markov Decision Processes: from Discrete to Continuous Optimization
• A Note on Optimization Formulations of Markov Decision Processes
• Explainable Deterministic MDPs
• Beyond Average Return in Markov Decision Processes