Portfolio Optimization under Recursive Utility via Reinforcement Learning

2026/03/24 by Minkey Chang · 1 voice
Computer Science · Economics, Econometrics and Finance · #cs.CE #q-fin.GN #q-fin.PM

paper · pdf · doi:10.48550/arxiv.2603.22880

Abstract

We study whether a risk-sensitive objective from asset-pricing theory -- recursive utility -- improves reinforcement learning for portfolio allocation. The Bellman equation under recursive utility involves a certainty equivalent (CE) of future value that has no closed form under observed returns; we approximate it by K-sample Monte Carlo and train actor-critic (PPO, A2C) on the resulting value target and an approximate advantage estimate (AAE) that generalizes the Bellman residual to multi-step with state-dependent weights. This formulation applies only to critic-based algorithms. On 10 chronological train/test splits of South Korean ETF data, the recursive-utility agent improves on the discounted (naive) baseline in Sharpe ratio, max drawdown, and cumulative return. Derivations, world model and metrics, and full result tables are in the appendices.

Citations

• FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance
• Proximal Policy Optimization Algorithms
• A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem
• Asynchronous Methods for Deep Reinforcement Learning
• High-Dimensional Continuous Control Using Generalized Advantage\n Estimation
• Substitution, Risk Aversion, and the Temporal Behavior of Consumption and Asset Returns: A Theoretical Framework
• PORTFOLIO SELECTION*
• Reinforcement Learning: An Introduction

Discussions

• Minkey Chang: Portfolio Optimization under Recursive Utility via Reinforcement Learning https://arxiv.org/abs/2603.22880 https://arxiv.org/pdf/2603.22880 https://arxiv.org/html/2603.22880 [bsky, 0 points, 0 comments]

Related

• Adaptive and Regime-Aware RL for Portfolio Optimization
• Asset Allocation: From Markowitz to Deep Reinforcement Learning
• Deep Reinforcement Learning and Convex Mean-Variance Optimisation for Portfolio Management
• Reinforcement Learning for Risk-Sensitive Investment Management: a Free Energy--Entropy Duality Approach
• Your Offline Policy is Not Trustworthy: Bilevel Reinforcement Learning for Sequential Portfolio Optimization