Foundations of Reinforcement Learning and Interactive Decision Making

2023/12/27 by Dylan J. Foster, Alexander Rakhlin, Foster, Dylan J. +1 · 2 voices · 6 citations
Computer Science · Engineering · Mathematics · #Advanced Control Systems Optimization #Artificial intelligence #Bayesian inference #Bayesian probability #COVID-19 epidemiological studies #Computer science #Data Stream Mining Techniques #Dilemma #Engineering #Epistemology #Frequentist inference #Function (biology) #Machine learning #Parallels #Perspective (graphical) #Reinforcement learning #Theme (computing) #cs.LG #math.OC #math.ST #stat.ML

paper · pdf · doi:10.48550/arxiv.2312.16730

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2023/12/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

These lecture notes give a statistical perspective on the foundations of reinforcement learning and interactive decision making. We present a unifying framework for addressing the exploration-exploitation dilemma using frequentist and Bayesian approaches, with connections and parallels between supervised learning/estimation and decision making as an overarching theme. Special attention is paid to function approximation and flexible model classes such as neural networks. Topics covered include multi-armed and contextual bandits, structured bandits, and reinforcement learning with high-dimensional feedback.

Cited by

• Statistical analysis of Inverse Entropy-regularized Reinforcement Learning
• Improved Training Mechanism for Reinforcement Learning via Online Model Selection
• SOMBRL: Scalable and Optimistic Model-Based RL
• Scalable Multi-Objective and Meta Reinforcement Learning via Gradient Estimation
• Learning to Answer from Correct Demonstrations
• Perturbing the Derivative: Wild Refitting for Model-Free Evaluation of Machine Learning Models under Bregman Losses

Discussions

• As my first post on this platform, allow me to advertise the RL theory lecture notes I have been developing with Sasha Rakhlin: arxiv.org/abs/2312.16730 (shameless repost of my pinned tweet) [bsky, 209 points, 4 comments]
• The algorithm here builds on the Decision-Estimation Coefficient ideas of Foster et al, which unify a large swath of bandit and reinforcement learning theory. Foster and Rakhlin have just put up their [bsky, 2 points, 0 comments]

Related

• Reinforcement Learning: An Overview
• Towards More Efficient, Robust, Instance-adaptive, and Generalizable Sequential Decision making
• A Comprehensive Survey of Reinforcement Learning: From Algorithms to Practical Challenges
• The Statistical Complexity of Interactive Decision Making
• A Survey of Exploration Methods in Reinforcement Learning