Doubly Robust Off-policy Value Evaluation for Reinforcement Learning

2015/11/11 by Jiang, Nan, Li, Lihong · 21 citations
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Methodology (stat.ME) #Systems and Control (eess.SY) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.1511.03722

Abstract

We study the problem of off-policy value evaluation in reinforcement learning (RL), where one aims to estimate the value of a new policy based on data collected by a different policy. This problem is often a critical step when applying RL in real-world problems. Despite its importance, existing general methods either have uncontrolled bias or suffer high variance. In this work, we extend the doubly robust estimator for bandits to sequential decision-making problems, which gets the best of both worlds: it is guaranteed to be unbiased and can have a much lower variance than the popular importance sampling estimators. We demonstrate the estimator's accuracy in several benchmark problems, and illustrate its use as a subroutine in safe policy improvement. We also provide theoretical results on the hardness of the problem, and show that our estimator can match the lower bound in certain scenarios.

Cited by

• Session-Level Optimization for Large-Scale Retrieval using REINFORCE with Multi-Step Off-Policy Correction
• Causal Judge Evaluation: Calibrated Surrogate Metrics for LLM Systems
• Auto-exploration for online reinforcement learning
• APRIL: Annotations for Policy evaluation with Reliable Inference from LLMs
• Experimentation Under Non-stationary Interference
• Confounding Robust Deep Reinforcement Learning: A Causal Approach
• Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning
• DeepEN: A Deep Reinforcement Learning Framework for Personalized Enteral Nutrition in Critical Care
• Offline Reinforcement Learning in Large State Spaces: Algorithms and Guarantees
• Anchored Supervised Fine-Tuning
• Evaluation-Aware Reinforcement Learning
• Test-Time Learning and Inference-Time Deliberation for Efficiency-First Offline Reinforcement Learning in Care Coordination and Population Health Management
• Pluralistic Off-policy Evaluation and Alignment
• Detecting Model Drifts in Non-Stationary Environment Using Edit Operation Measures
• Hybrid Adaptive Conformal Offline Reinforcement Learning for Fair Population Health Management
• Incentivizing Safer Actions in Policy Optimization for Constrained Reinforcement Learning
• Beyond Prediction: Reinforcement Learning as the Defining Leap in Healthcare AI
• MORE-CLEAR: Multimodal Offline Reinforcement learning for Clinical notes Leveraged Enhanced State Representation
• Central Limit Theorems for Transition Probabilities of Controlled Markov Chains
• PERRY: Policy Evaluation with Confidence Intervals using Auxiliary Data
• A Two-armed Bandit Framework for A/B Testing

Related

• More Robust Doubly Robust Off-policy Evaluation
• Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learning
• Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning
• Robust On-Policy Sampling for Data-Efficient Policy Evaluation in Reinforcement Learning
• Doubly Optimal Policy Evaluation for Reinforcement Learning