Control randomisation approach for policy gradient and application to reinforcement learning in optimal switching

2024/04/27 by Robert Denkert, Huyên Pham, Denkert, Robert +3 · 3 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · Engineering · #68T07 (Secondary) #93E20 (Primary) #Electric Vehicles and Infrastructure #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (stat.ML) #Optimization and Control (math.OC) #Reinforcement Learning in Robotics #Viral Infectious Diseases and Gene Expression in Insects

paper · pdf · doi:10.48550/arxiv.2404.17939

openalex publication_date 2024/04/27 · openalex created_date 2024/05/11 · openalex updated_date 2026/07/28

Abstract

We propose a comprehensive framework for policy gradient methods tailored to continuous time reinforcement learning. This is based on the connection between stochastic control problems and randomised problems, enabling applications across various classes of Markovian continuous time control problems, beyond diffusion models, including e.g. regular, impulse and optimal stopping/switching problems. By utilizing change of measure in the control randomisation technique, we derive a new policy gradient representation for these randomised problems, featuring parametrised intensity policies. We further develop actor-critic algorithms specifically designed to address general Markovian stochastic control issues. Our framework is demonstrated through its application to optimal switching problems, with two numerical case studies in the energy sector focusing on real options.

Cited by

• Continuous-time reinforcement learning for optimal switching over multiple regimes
• Entropy Regularization in Mean-Field Games of Optimal Stopping
• A Two-fold Randomization Framework for Impulse Control Problems

Related

• Continuous-time reinforcement learning for optimal switching over multiple regimes
• Learning Optimal Deterministic Policies with Stochastic Policy Gradients
• Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms
• Actor-Critic learning for mean-field control in continuous time
• Distributional Policy Optimization: An Alternative Approach for Continuous Control