Model-based Offline Reinforcement Learning with Count-based Conservatism

2023/07/21 by Byeongchan Kim, Kim, Byeongchan, Min-hwan Oh +1 · 1 citation
Computer Science · Engineering · #FOS: Computer and information sciences #Fuel Cells and Related Materials #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2307.11352

openalex publication_date 2023/07/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we propose a model-based offline reinforcement learning method that integrates count-based conservatism, named Count-MORL. Our method utilizes the count estimates of state-action pairs to quantify model estimation error, marking the first algorithm of demonstrating the efficacy of count-based conservatism in model-based offline deep RL to the best of our knowledge. For our proposed method, we first show that the estimation error is inversely proportional to the frequency of state-action pairs. Secondly, we demonstrate that the learned policy under the count-based conservative model offers near-optimality performance guarantees. Through extensive numerical experiments, we validate that Count-MORL with hash code implementation significantly outperforms existing offline RL algorithms on the D4RL benchmark datasets. The code is accessible at \hrefhttps://github.com/oh-lab/Count-MORLhttps://github.com/oh-lab/Count-MORL.

Cited by

• Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatism

Related

• DOMAIN: MilDly COnservative Model-BAsed OfflINe Reinforcement Learning
• Offline Reinforcement Learning with Reverse Model-based Imagination
• Compositional Conservatism: A Transductive Approach in Offline Reinforcement Learning
• Mildly Conservative Regularized Evaluation for Offline Reinforcement Learning
• Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatism