Dimension-Wise Importance Sampling Weight Clipping for Sample-Efficient\n Reinforcement Learning

2019/05/07 by Seungyul Han, Han, Seungyul, Youngchul Sung +1 · 3 citations
Computer Science · Decision Sciences · #Reinforcement Learning in Robotics #Advanced Bandit Algorithms Research #Adaptive Dynamic Programming Control

paper · pdf · doi:10.48550/arxiv.1905.02363

Abstract

In importance sampling (IS)-based reinforcement learning algorithms such as\nProximal Policy Optimization (PPO), IS weights are typically clipped to avoid\nlarge variance in learning. However, policy update from clipped statistics\ninduces large bias in tasks with high action dimensions, and bias from clipping\nmakes it difficult to reuse old samples with large IS weights. In this paper,\nwe consider PPO, a representative on-policy algorithm, and propose its\nimprovement by dimension-wise IS weight clipping which separately clips the IS\nweight of each action dimension to avoid large bias and adaptively controls the\nIS weight to bound policy update from the current policy. This new technique\nenables efficient learning for high action-dimensional tasks and reusing of old\nsamples like in off-policy learning to increase the sample efficiency.\nNumerical results show that the proposed new algorithm outperforms PPO and\nother RL algorithms in various Open AI Gym tasks.\n

Cited by

• Diversity Actor-Critic: Sample-Aware Entropy Regularization for\n Sample-Efficient Exploration
• Beyond Fixed Morphologies: Learning Graph Policies with Trust Region Compensation in Variable Action Spaces
• Intelligent Control of Spacecraft Reaction Wheel Attitude Using Deep Reinforcement Learning

Related

• On the Reuse Bias in Off-Policy Reinforcement Learning
• Weight Clipping for Deep Continual and Reinforcement Learning
• Relative Importance Sampling for off-Policy Actor-Critic in Deep Reinforcement Learning
• Importance Resampling for Off-policy Prediction
• Value-aware Importance Weighting for Off-policy Reinforcement Learning