Conservative Exploration in Reinforcement Learning

2020/02/08 by Evrard Garcelon, Garcelon, Evrard, Mohammad Ghavamzadeh +5
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2002.03218

openalex publication_date 2020/02/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

While learning in an unknown Markov Decision Process (MDP), an agent should trade off exploration to discover new information about the MDP, and exploitation of the current knowledge to maximize the reward. Although the agent will eventually learn a good or optimal policy, there is no guarantee on the quality of the intermediate policies. This lack of control is undesired in real-world applications where a minimum requirement is that the executed policies are guaranteed to perform at least as well as an existing baseline. In this paper, we introduce the notion of conservative exploration for average reward and finite horizon problems. We present two optimistic algorithms that guarantee (w.h.p.) that the conservative constraint is never violated during learning. We derive regret bounds showing that being conservative does not hinder the learning ability of these algorithms.

Citations

• Constrained Markov Decision Processes
• REGAL: A Regularization based Algorithm for Reinforcement Learning in Weakly Communicating MDPs
• Counterfactual Risk Minimization: Learning from Logged Bandit Feedback
• Conservative Contextual Linear Bandits
• Safe Model-based Reinforcement Learning with Stability Guarantees
• Provably Efficient Reinforcement Learning with Linear Function Approximation

Related

• Near-optimal Conservative Exploration in Reinforcement Learning under Episode-wise Constraints
• Conservative Exploration for Policy Optimization via Off-Policy Policy Evaluation
• DOPE: Doubly Optimistic and Pessimistic Exploration for Safe Reinforcement Learning
• Learning Policies with Zero or Bounded Constraint Violation for\n Constrained MDPs
• Uniformly Conservative Exploration in Reinforcement Learning