Fully Offline Reinforcement Learning

2025/05/28 by Mattie Fellows, Clarisse Wibault, Fellows, Mattie +11 · 1 voice
Engineering · Computer Science · #Elevator Systems and Control #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2505.22442

Abstract

Offline RL (ORL) promises safe and sample-efficient deployment but existing methods rely on undocumented online interactions for hyperparameter tuning and lack reliable fully offline estimates of initial online performance. We introduce SOReL, a fully offline Bayesian model-based RL method that learns a posterior over dynamics, estimates policy value via predictive uncertainty, and enables complete offline hyperparameter selection. We further propose TOReL, which extends this tuning framework to arbitrary model-free and model-based ORL algorithms. We provide a regret analysis showing that Bayesian offline RL achieves the minimax-optimal parametric rate under standard regularity conditions. Together, our methods establish a practical and theoretically grounded framework for fully offline RL.

Discussions

• 2/2 🚀 Our new paper below tackles two major issues of high online sample complexity and lack of online performance guarantees in offline RL, obtaining accurate regret estimation and achieving competi [bsky, 4 points, 0 comments]

Related

• Bayesian Design Principles for Offline-to-Online Reinforcement Learning
• A Minimalist Approach to Offline Reinforcement Learning
• MOReL : Model-Based Offline Reinforcement Learning
• A Clean Slate for Offline Reinforcement Learning
• You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL