Pessimism for Offline Linear Contextual Bandits using ℓ_p Confidence Sets

2022/05/21 by Li, Gene, Ma, Cong, Srebro, Nathan · 4 citations
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.2205.10671

Abstract

We present a family \π\_{p≥ 1} of pessimistic learning rules for offline learning of linear contextual bandits, relying on confidence sets with respect to different ℓ_p norms, where π₂ corresponds to Bellman-consistent pessimism (BCP), while π_∞ is a novel generalization of lower confidence bound (LCB) to the linear setting. We show that the novel π_∞ learning rule is, in a sense, adaptively optimal, as it achieves the minimax performance (up to log factors) against all ℓ_q-constrained problems, and as such it strictly dominates all other predictors in the family, including π₂.

Cited by

• Hybrid Combinatorial Multi-armed Bandits with Probabilistically Triggered Arms
• Group-Sensitive Offline Contextual Bandits
• KL-regularization Itself is Differentially Private in Bandits and RLHF
• Augmenting Online RL with Offline Data is All You Need: A Unified Hybrid RL Algorithm Design and Analysis

Related

• Upper Counterfactual Confidence Bounds: a New Optimism Principle for Contextual Bandits
• Bellman-consistent Pessimism for Offline Reinforcement Learning
• Oracle-Efficient Pessimism: Offline Policy Optimization in Contextual Bandits
• Logarithmic Smoothing for Pessimistic Off-Policy Evaluation, Selection and Learning
• Rectified Pessimistic-Optimistic Learning for Stochastic Continuum-armed Bandit with Constraints