Policy Optimization in a Noisy Neighborhood: On Return Landscapes in Continuous Control

2023/09/26 by Nate Rahn, Pierluca D’Oro, Rahn, Nate +9 · 1 voice · 2 citations
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #Auction Theory and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #cs.LG

paper · pdf · doi:10.48550/arxiv.2309.14597

openalex publication_date 2023/09/26 · arxiv published 2023/09/26 · openalex created_date 2023/09/29 · arxiv updated 2024/04/10 · openalex updated_date 2026/07/28

Abstract

Deep reinforcement learning agents for continuous control are known to exhibit significant instability in their performance over time. In this work, we provide a fresh perspective on these behaviors by studying the return landscape: the mapping between a policy and a return. We find that popular algorithms traverse noisy neighborhoods of this landscape, in which a single update to the policy parameters leads to a wide range of returns. By taking a distributional view of these returns, we map the landscape, characterizing failure-prone regions of policy space and revealing a hidden dimension of policy quality. We show that the landscape exhibits surprising structure by finding simple paths in parameter space which improve the stability of a policy. To conclude, we develop a distribution-aware procedure which finds such paths, navigating away from noisy neighborhoods in order to improve the robustness of a policy. Taken together, our results provide new insight into the optimization, evaluation, and design of agents.

Cited by

• Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learning
• Efficient On-Policy Reinforcement Learning via Exploration of Sparse Parameter Space

Discussions

• 📝 Policy Optimization in a Noisy Neighborhood: On Return Landscapes in Continuous Control 🧠 "Studies the return landscape and find it has surprising structure: there are simple paths in parameter sp [mastodon, 0 points, 0 comments]

Related

• Action Noise in Off-Policy Deep Reinforcement Learning: Impact on Exploration and Performance
• Robust Policy Optimization in Deep Reinforcement Learning
• Exploring More When It Needs in Deep Reinforcement Learning
• Policy Search by Target Distribution Learning for Continuous Control
• The Definitive Guide to Policy Gradients in Deep Reinforcement Learning: Theory, Algorithms and Implementations