Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach

2025/05/10 by Minting Pan, Yitao Zheng, Pan, Minting +7 · 1 citation
Computer Science · Engineering · #Reinforcement Learning in Robotics #Robot Manipulation and Learning #Adversarial Robustness in Machine Learning

paper · pdf · doi:10.48550/arxiv.2505.06482

Abstract

Offline reinforcement learning (RL) enables policy optimization using static datasets, avoiding the risks and costs of extensive real-world exploration. However, it struggles with suboptimal offline behaviors and inaccurate value estimation due to the lack of environmental interaction. We present Video-Enhanced Offline RL (VeoRL), a model-based method that constructs an interactive world model from diverse, unlabeled video data readily available online. Leveraging model-based behavior guidance, our approach transfers commonsense knowledge of control policy and physical dynamics from natural videos to the RL agent within the target domain. VeoRL achieves substantial performance gains (over 100% in some cases) across visual control tasks in robotic manipulation, autonomous driving, and open-world video games.

Cited by

• Planning with Reasoning using Vision Language World Model

Related

• Real-World Offline Reinforcement Learning from Vision Language Model Feedback
• A Workflow for Offline Model-Free Robotic Reinforcement Learning
• Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretraining
• Robotic Offline RL from Internet Videos via Value-Function Pre-Training
• Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learning