Doubly Robust Off-policy Value Evaluation for Reinforcement Learning

2015/11/11 by Nan Jiang, Jiang, Nan, Lihong Li +1 · 38 citations
Computer Science · Decision Sciences · Mathematics · #Advanced Bandit Algorithms Research #Advanced Causal Inference Techniques #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Methodology (stat.ME) #Reinforcement Learning in Robotics #Systems and Control (eess.SY) #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.1511.03722

openalex publication_date 2015/11/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We study the problem of off-policy value evaluation in reinforcement learning (RL), where one aims to estimate the value of a new policy based on data collected by a different policy. This problem is often a critical step when applying RL in real-world problems. Despite its importance, existing general methods either have uncontrolled bias or suffer high variance. In this work, we extend the doubly robust estimator for bandits to sequential decision-making problems, which gets the best of both worlds: it is guaranteed to be unbiased and can have a much lower variance than the popular importance sampling estimators. We demonstrate the estimator's accuracy in several benchmark problems, and illustrate its use as a subroutine in safe policy improvement. We also provide theoretical results on the hardness of the problem, and show that our estimator can match the lower bound in certain scenarios.

Citations

• Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score

Cited by

• Session-Level Optimization for Large-Scale Retrieval using REINFORCE with Multi-Step Off-Policy Correction
• Causal Judge Evaluation: Calibrated Surrogate Metrics for LLM Systems
• Auto-exploration for online reinforcement learning
• APRIL: Annotations for Policy evaluation with Reliable Inference from LLMs
• Experimentation Under Non-stationary Interference
• Confounding Robust Deep Reinforcement Learning: A Causal Approach
• Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning
• DeepEN: A Deep Reinforcement Learning Framework for Personalized Enteral Nutrition in Critical Care
• Offline Reinforcement Learning in Large State Spaces: Algorithms and Guarantees
• Anchored Supervised Fine-Tuning
• Evaluation-Aware Reinforcement Learning
• Test-Time Learning and Inference-Time Deliberation for Efficiency-First Offline Reinforcement Learning in Care Coordination and Population Health Management
• Pluralistic Off-policy Evaluation and Alignment
• Detecting Model Drifts in Non-Stationary Environment Using Edit Operation Measures
• Hybrid Adaptive Conformal Offline Reinforcement Learning for Fair Population Health Management
• Incentivizing Safer Actions in Policy Optimization for Constrained Reinforcement Learning
• Beyond Prediction: Reinforcement Learning as the Defining Leap in Healthcare AI
• MORE-CLEAR: Multimodal Offline Reinforcement learning for Clinical notes Leveraged Enhanced State Representation
• Central Limit Theorems for Transition Probabilities of Controlled Markov Chains
• PERRY: Policy Evaluation with Confidence Intervals using Auxiliary Data
• A Two-armed Bandit Framework for A/B Testing
• Off-Policy Evaluation Under Nonignorable Missing Data
• Sim2Val: Leveraging Correlation Across Test Platforms for Variance-Reduced Metric Estimation
• WorldGym: World Model as An Environment for Policy Evaluation
• Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluation
• A Bayesian Nonparametric Approach for Estimating Individualized Treatment-Response Curves
• STITCH-OPE: Trajectory Stitching with Guided Diffusion for Off-Policy Evaluation
• Balancing Interference and Correlation in Spatial Experimental Designs: A Causal Graph Cut Approach
• Is Deep Reinforcement Learning Ready for Practical Applications in Healthcare? A Sensitivity Analysis of Duel-DDQN for Hemodynamic Management in Sepsis Patients
• Treatment Effect Estimation for Optimal Decision-Making
• Task Selection Policies for Multitask Learning
• Automatic Reward Shaping from Confounded Offline Data
• DOLCE: Decomposing Off-Policy Evaluation/Learning into Lagged and Current Effects
• Smooth Multi-Policy Causal Effect Estimation in Longitudinal Settings
• Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic
• Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning
• Representation and Reinforcement Learning for Personalized Glycemic Control in Septic Patients
• Behaviour Policy Estimation in Off-Policy Policy Evaluation: Calibration Matters

Related

• More Robust Doubly Robust Off-policy Evaluation
• Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learning
• Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning
• Robust On-Policy Sampling for Data-Efficient Policy Evaluation in Reinforcement Learning
• Doubly Optimal Policy Evaluation for Reinforcement Learning