Conservative Optimistic Policy Optimization via Multiple Importance Sampling

2021/03/04 by Achraf Azize, Azize, Achraf, Othman Gaizi +1
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Optimization and Search Problems #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2103.03307

openalex publication_date 2021/03/04 · openalex created_date 2021/03/15 · openalex updated_date 2026/07/28

Abstract

Reinforcement Learning (RL) has been able to solve hard problems such as playing Atari games or solving the game of Go, with a unified approach. Yet modern deep RL approaches are still not widely used in real-world applications. One reason could be the lack of guarantees on the performance of the intermediate executed policies, compared to an existing (already working) baseline policy. In this paper, we propose an online model-free algorithm that solves conservative exploration in the policy optimization problem. We show that the regret of the proposed approach is bounded by O(√(T)) for both discrete and continuous parameter spaces.

Related

• Conservative Exploration for Policy Optimization via Off-Policy Policy Evaluation
• Optimistic Natural Policy Gradient: a Simple Efficient Policy Optimization Framework for Online RL
• Conservative Dual Policy Optimization for Efficient Model-Based Reinforcement Learning
• CROP: Conservative Reward for Model-based Offline Policy Optimization
• Clipped-Objective Policy Gradients for Pessimistic Policy Optimization