Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning

2019/11/15 by Cameron Voloshin, Voloshin, Cameron, Hoang Le +6 · 22 citations
Computer Science · Mathematics · #Formal Methods in Verification #Reinforcement Learning in Robotics #Software Reliability and Analysis Research #cs.AI #cs.LG #cs.RO #stat.ML

paper · pdf · doi:10.48550/arxiv.1911.06854

arxiv created 2021/11/27 · arxiv updated 2021/11/30

Abstract

We offer an experimental benchmark and empirical study for off-policy policy evaluation (OPE) in reinforcement learning, which is a key problem in many safety critical applications. Given the increasing interest in deploying learning-based methods, there has been a flurry of recent proposals for OPE method, leading to a need for standardized empirical analyses. Our work takes a strong focus on diversity of experimental design to enable stress testing of OPE methods. We provide a comprehensive benchmarking suite to study the interplay of different attributes on method performance. We distill the results into a summarized set of guidelines for OPE in practice. Our software package, the Caltech OPE Benchmarking Suite (COBS), is open-sourced and we invite interested researchers to further contribute to the benchmark.

Citations

• Adjustment of an Inverse Matrix Corresponding to a Change in One Element of a Given Matrix
• Doubly Robust Estimation in Missing Data and Causal Inference Models

Cited by

• Minimax Value Interval for Off-Policy Evaluation and Policy Optimization
• Active Offline Policy Selection
• Benchmarking Offline Multi-Objective Reinforcement Learning in Critical Care
• Breaking Determinism: Stochastic Modeling for Reliable Off-Policy Evaluation in Ad Auctions
• A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representation
• Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings
• Guardian-regularized Safe Offline Reinforcement Learning for Smart Weaning of Mechanical Circulatory Devices
• A Workflow for Offline Model-Free Robotic Reinforcement Learning
• Benchmarks for Deep Off-Policy Evaluation
• Learning Deep Features in Instrumental Variable Regression
• Perturbing the Derivative: Wild Refitting for Model-Free Evaluation of Machine Learning Models under Bregman Losses
• Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learning
• Bootstrapping Fitted Q-Evaluation for Off-Policy Inference
• Off-Policy Evaluation Under Nonignorable Missing Data
• Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach
• Off-Policy Evaluation in Partially Observed Markov Decision Processes under Sequential Ignorability
• Adaptive Estimator Selection for Off-Policy Evaluation
• Offline RL Without Off-Policy Evaluation
• STITCH-OPE: Trajectory Stitching with Guided Diffusion for Off-Policy Evaluation
• Autoregressive Dynamics Models for Offline Policy Evaluation and\n Optimization
• NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
• DOLCE: Decomposing Off-Policy Evaluation/Learning into Lagged and Current Effects
• Smooth Multi-Policy Causal Effect Estimation in Longitudinal Settings

Related

• A Review of Off-Policy Evaluation in Reinforcement Learning
• Safe Evaluation For Offline Learning: Are We Ready To Deploy?
• Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning
• Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol
• When is Offline Policy Selection Sample Efficient for Reinforcement Learning?