A Pontryagin Perspective on Reinforcement Learning

2024/05/28 by Onno Eberhard, Claire Vernade, Eberhard, Onno +3 · 1 voice · 2 citations
#cs.LG #math.OC

paper · pdf · doi:10.48550/arxiv.2405.18100

Abstract

Reinforcement learning has traditionally focused on learning state-dependent policies to solve optimal control problems in a closed-loop fashion. In this work, we introduce the paradigm of open-loop reinforcement learning where a fixed action sequence is learned instead. We present three new algorithms: one robust model-based method and two sample-efficient model-free methods. Rather than basing our algorithms on Bellman's equation from dynamic programming, our work builds on Pontryagin's principle from the theory of open-loop optimal control. We provide convergence guarantees and evaluate all methods empirically on a pendulum swing-up task, as well as on two high-dimensional MuJoCo tasks, significantly outperforming existing baselines.

Cited by

• Neural Co-state Policies: Structuring Hidden States in Recurrent Reinforcement Learning
• Continuous-Time Value Iteration for Multi-Agent Reinforcement Learning

Discussions

• A bit rusty with Control, and found this by accident, but quite enjoyed reading it! arxiv.org/abs/2405.18100 [bsky, 5 points, 0 comments]

Related

• A General Control-Theoretic Approach for Reinforcement Learning: Theory and Algorithms
• A Crash Course on Reinforcement Learning
• A Tour of Reinforcement Learning: The View from Continuous Control
• On the Search for Feedback in Reinforcement Learning
• A General Markov Decision Process Framework for Directly Learning Optimal Control Policies