DOLCE: Decomposing Off-Policy Evaluation/Learning into Lagged and Current Effects

2025/05/02 by Shu Tamano, Tamano, Shu, Nojima, Masanori
Decision Sciences · #Action (physics) #Context (archaeology) #Current (fluid) #Evaluation and Performance Assessment #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Residual #Term (time) #Value (mathematics)

paper · pdf · doi:10.48550/arxiv.2505.00961

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/05/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Off-policy evaluation and learning in contextual bandits use logged interaction data to estimate and optimize the value of a target policy. Most existing methods require sufficient action overlap between the logging and target policies, and violations can bias value and policy gradient estimates. To address this issue, we propose DOLCE (Decomposing Off-policy evaluation/learning into Lagged and Current Effects), which uses only lagged contexts already stored in bandit logs to construct lag-marginalized importance weights and to decompose the objective into a support-robust lagged correction term and a current, model-based term, yielding bias cancellation when the reward-model residual is conditionally mean-zero given the lagged context and action. With multiple candidate lags, DOLCE softly aggregates lag-specific estimates, and we introduce a moment-based training procedure that promotes the desired invariance using only logged lag-augmented data. We show that DOLCE is unbiased in an idealized setting and yields consistent and asymptotically normal estimates with cross-fitting under standard conditions. Our experiments demonstrate that DOLCE achieves substantial improvements in both off-policy evaluation and learning, particularly as the proportion of individuals who violate support increases.

Citations

• Policy Learning for Off-Dynamics RL with Deficient Support
• POTEC: Off-Policy Learning for Large Action Spaces via Two-Stage Policy Decomposition
• Towards Assessing and Benchmarking Risk-Return Tradeoff of Off-Policy Evaluation
• Positivity-free Policy Learning with Observational Data
• Average treatment effect on the treated, under lack of positivity
• Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modeling
• Policy learning "without" overlap: Pessimism and generalized empirical Bernstein's inequality
• Causal Estimation for Text Data with (Apparent) Overlap Violations
• Off-Policy Evaluation for Large Action Spaces via Embeddings
• Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble
• Optimal Off-Policy Evaluation from Multiple Logging Policies
• Conservative Q-Learning for Offline Reinforcement Learning
• Off-Policy Estimation of Long-Term Average Outcomes With Applications to Mobile Health
• Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning
• Understanding the Curse of Horizon in Off-Policy Evaluation via Conditional Importance Sampling
• Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes
• Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Off-Policy Deep Reinforcement Learning without Exploration
• Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
• More Robust Doubly Robust Off-policy Evaluation
• Overlap in Observational Studies with High-Dimensional Covariates
• Nonparametric causal effects based on incremental propensity score interventions
• Off-policy evaluation for slate recommendation
• Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning
• Doubly Robust Off-policy Value Evaluation for Reinforcement Learning
• Balancing Covariates via Propensity Score Weighting
• Counterfactual Reasoning and Learning Systems
• Doubly Robust Policy Evaluation and Learning
• Diagnosing and responding to violations in the positivity assumption
• Doubly Robust Estimation in Missing Data and Causal Inference Models
• Estimating causal effects of treatments in randomized and nonrandomized studies.

Related

• Off-Policy Evaluation Using Information Borrowing and Context-Based Switching
• Cross-Domain Off-Policy Evaluation and Learning for Contextual Bandits
• Doubly Robust Policy Evaluation and Learning
• A General Framework for Off-Policy Learning with Partially-Observed Reward
• Counterfactual Learning with General Data-generating Policies