Importance Sampling based Exploration in Q Learning

2021/07/01 by Vijay Kumar, Kumar, Vijay, Mort Webster +1
Computer Science · Mathematics · #Data Stream Mining Techniques #FOS: Mathematics #Face and Expression Recognition #Optimization and Control (math.OC) #Reinforcement Learning in Robotics #math.OC

paper · pdf · doi:10.48550/arxiv.2107.00602

arxiv created 2021/07/01 · openalex publication_date 2021/07/01 · arxiv updated 2021/07/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Approximate Dynamic Programming (ADP) is a methodology to solve multi-stage stochastic optimization problems in multi-dimensional discrete or continuous spaces. ADP approximates the optimal value function by adaptively sampling both action and state space. It provides a tractable approach to very large problems, but can suffer from the exploration-exploitation dilemma. We propose a novel approach for selecting actions using importance sampling weighted by the value function approximation in continuous decision spaces to address this dilemma. An advantage of this approach is it balances exploration and exploitation without any tuning parameters when sampling actions compared to other exploration approaches such as Epsilon Greedy, instead relying only on the approximate value function. We compare the proposed algorithm with other exploration strategies in continuous action space in the context of a multi-stage generation expansion planning problem under uncertainty.

Citations

• Scenarios and Policy Aggregation in Optimization Under Uncertainty
• Continuous control with deep reinforcement learning
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Count-Based Exploration with Neural Density Models
• Using Options and Covariance Testing for Long Horizon Off-Policy Policy Evaluation
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Stochastic dual dynamic integer programming
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Q-Learning for Continuous Actions with Cross-Entropy Guided Policies
• CAQL: Continuous Action Q-Learning

Related

• Probabilistic Exploration in Planning while Learning
• Marginal Utility for Planning in Continuous or Large Discrete Action Spaces
• Approximate Dynamic Programming By Minimizing Distributionally Robust Bounds
• A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representation
• Conservative Optimistic Policy Optimization via Multiple Importance Sampling