Reinforcement Learning with Dynamic Convex Risk Measures

2021/12/26 by Anthony Coache, Coache, Anthony, Sebastian Jaimungal +1 · 3 citations
Decision Sciences · Energy · Economics, Econometrics and Finance · #Risk and Portfolio Optimization #Energy, Environment, and Transportation Policies #Stochastic processes and financial applications

paper · pdf · doi:10.48550/arxiv.2112.13414

Abstract

We develop an approach for solving time-consistent risk-sensitive stochastic optimization problems using model-free reinforcement learning (RL). Specifically, we assume agents assess the risk of a sequence of random variables using dynamic convex risk measures. We employ a time-consistent dynamic programming principle to determine the value of a particular policy, and develop policy gradient update rules that aid in obtaining optimal policies. We further develop an actor-critic style algorithm using neural networks to optimize over policies. Finally, we demonstrate the performance and flexibility of our approach by applying it to three optimization problems: statistical arbitrage trading strategies, financial hedging, and obstacle avoidance robot control.

Cited by

• Online Policy Evaluation for MDPs with Dynamic UBSR Measures
• Learning functions through Diffusion Maps
• Periodic evaluation of defined-contribution pension fund: A dynamic risk measure approach

Related

• Risk-sensitive Reinforcement Learning Based on Convex Scoring Functions
• A Convex Programming Approach to Data-Driven Risk-Averse Reinforcement Learning
• A policy gradient approach for optimization of smooth risk measures
• Robust Reinforcement Learning with Dynamic Distortion Risk Measures
• Conditionally Elicitable Dynamic Risk Measures for Deep Reinforcement Learning