Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learning

2021/10/26 by Siyuan Zhang, Nan Jiang, Zhang, Siyuan +1 · 2 citations
Computer Science · Engineering · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics #VLSI and FPGA Design Techniques

paper · pdf · doi:10.48550/arxiv.2110.14000

openalex publication_date 2021/10/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

How to select between policies and value functions produced by different training algorithms in offline reinforcement learning (RL) -- which is crucial for hyperpa-rameter tuning -- is an important open question. Existing approaches based on off-policy evaluation (OPE) often require additional function approximation and hence hyperparameters, creating a chicken-and-egg situation. In this paper, we design hyperparameter-free algorithms for policy selection based on BVFT [XJ21], a recent theoretical advance in value-function selection, and demonstrate their effectiveness in discrete-action benchmarks such as Atari. To address performance degradation due to poor critics in continuous-action domains, we further combine BVFT with OPE to get the best of both worlds, and obtain a hyperparameter-tuning method for Q-function based OPE with theoretical guarantees as a side product.

Citations

• Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
• A Kernel Loss for Solving the Bellman Equation
• Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• MOReL : Model-Based Offline Reinforcement Learning
• Is Pessimism Provably Efficient for Offline RL?
• Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency
• Bellman-consistent Pessimism for Offline Reinforcement Learning

Cited by

• Selecting Belief-State Approximations in Simulators with Latent States
• Offline Reinforcement Learning in Large State Spaces: Algorithms and Guarantees

Related

• Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol
• Hyperparameter Selection for Offline Reinforcement Learning
• When is Offline Policy Selection Sample Efficient for Reinforcement Learning?
• Hyperparameter Selection Methods for Fitted Q-Evaluation with Error Guarantee
• Offline RL Without Off-Policy Evaluation