Deep Exploration via Randomized Value Functions

2017/03/22 by Ian Osband, Benjamin Van Roy, Osband, Ian +5 · 31 citations
Computer Science · Decision Sciences · Mathematics · #Advanced Bandit Algorithms Research #Adversarial Robustness in Machine Learning #Reinforcement Learning in Robotics #cs.AI #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1703.07608

Accepted for publication in Journal of Machine Learning Research 2019

arxiv created 2019/09/23 · arxiv updated 2019/09/25

Abstract

We study the use of randomized value functions to guide deep exploration in reinforcement learning. This offers an elegant means for synthesizing statistically and computationally efficient exploration with common practical approaches to value function learning. We present several reinforcement learning algorithms that leverage randomized value functions and demonstrate their efficacy through computational studies. We also prove a regret bound that establishes statistical efficiency with a tabular representation.

Citations

• The Efficiency Analysis of Choices Involving Risk
• Thompson Sampling for Contextual Bandits with Linear Payoffs
• On Lower Bounds for Regret in Reinforcement Learning

Cited by

• The Uncertainty Bellman Equation and Exploration
• Variational Bayesian Reinforcement Learning with Regret Bounds
• Tensor-Efficient High-Dimensional Q-learning
• Value of Information-Enhanced Exploration in Bootstrapped DQN
• Noisy Networks for Exploration
• TTR-Based Reward for Reinforcement Learning with Implicit Model Priors
• Worst-Case Regret Bounds for Exploration via Randomized Value Functions
• Representation-Based Exploration for Language Models: From Test-Time to Post-Training
• Diversity-Incentivized Exploration for Versatile Reasoning
• Functional Critic Modeling for Provably Convergent Off-Policy Actor-Critic
• On the Limits of Tabular Hardness Metrics for Deep RL: A Study with the Pharos Benchmark
• Model-Based Reinforcement Learning with Value-Targeted Regression
• Why is Posterior Sampling Better than Optimism for Reinforcement Learning?
• Non-local Policy Optimization via Diversity-regularized Collaborative Exploration
• Parameterized Indexed Value Function for Efficient Exploration in Reinforcement Learning
• Exploitation Is All You Need... for Exploration
• Implicit Generative Modeling for Efficient Exploration
• A Short Survey on Probabilistic Reinforcement Learning
• Uncertainty-sensitive Learning and Planning with Ensembles
• Q-learning with Posterior Sampling
• Convex Q-Learning, Part 1: Deterministic Optimal Control
• Universal Value-Function Uncertainties
• Behaviour Suite for Reinforcement Learning
• Periodic Intra-Ensemble Knowledge Distillation for Reinforcement Learning
• Generalization in Monitored Markov Decision Processes (Mon-MDPs)
• On the Sample Complexity of Reinforcement Learning with Policy Space Generalization
• Genetic-Gated Networks for Deep Reinforcement
• Bayesian Neural Network Ensembles
• Behavior-Consistent Deep Reinforcement Learning
• On the Equivalence of Random Network Distillation, Deep Ensembles, and Bayesian Inference
• Toward Efficient Exploration by Large Language Model Agents
• Mathematical methods of reinforcement learning

Related

• Worst-Case Regret Bounds for Exploration via Randomized Value Functions
• Generalization and Exploration via Randomized Value Functions
• Randomized Exploration for Reinforcement Learning with General Value Function Approximation
• On Optimistic versus Randomized Exploration in Reinforcement Learning
• Parameterized Indexed Value Function for Efficient Exploration in Reinforcement Learning