Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning

2019/11/15 by Cameron Voloshin, Voloshin, Cameron, Hoang Le +5 · 13 citations
Computer Science · #Reinforcement Learning in Robotics #Software Reliability and Analysis Research #Formal Methods in Verification

paper · pdf · doi:10.48550/arxiv.1911.06854

Abstract

We offer an experimental benchmark and empirical study for off-policy policy evaluation (OPE) in reinforcement learning, which is a key problem in many safety critical applications. Given the increasing interest in deploying learning-based methods, there has been a flurry of recent proposals for OPE method, leading to a need for standardized empirical analyses. Our work takes a strong focus on diversity of experimental design to enable stress testing of OPE methods. We provide a comprehensive benchmarking suite to study the interplay of different attributes on method performance. We distill the results into a summarized set of guidelines for OPE in practice. Our software package, the Caltech OPE Benchmarking Suite (COBS), is open-sourced and we invite interested researchers to further contribute to the benchmark.

Citations

• Adjustment of an Inverse Matrix Corresponding to a Change in One Element of a Given Matrix
• Doubly Robust Estimation in Missing Data and Causal Inference Models

Cited by

• Minimax Value Interval for Off-Policy Evaluation and Policy Optimization
• Active Offline Policy Selection
• Benchmarking Offline Multi-Objective Reinforcement Learning in Critical Care
• Breaking Determinism: Stochastic Modeling for Reliable Off-Policy Evaluation in Ad Auctions
• A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representation
• Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings
• Guardian-regularized Safe Offline Reinforcement Learning for Smart Weaning of Mechanical Circulatory Devices
• A Workflow for Offline Model-Free Robotic Reinforcement Learning
• Benchmarks for Deep Off-Policy Evaluation
• Learning Deep Features in Instrumental Variable Regression
• Perturbing the Derivative: Wild Refitting for Model-Free Evaluation of Machine Learning Models under Bregman Losses
• Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learning
• Bootstrapping Fitted Q-Evaluation for Off-Policy Inference

Related

• A Review of Off-Policy Evaluation in Reinforcement Learning
• Safe Evaluation For Offline Learning: Are We Ready To Deploy?
• Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning
• Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol
• When is Offline Policy Selection Sample Efficient for Reinforcement Learning?