Diffusion Reward: Learning Rewards via Conditional Video Diffusion

2023/12/21 by Tao Huang, Guangqi Jiang, Huang, Tao +5 · 15 citations
Computer Science · #Artificial intelligence #Code (set theory) #Computer science #Conditional entropy #Diffusion #Entropy (arrow of time) #Generative grammar #Key (lock) #Machine learning #Principle of maximum entropy #Reinforcement Learning in Robotics #Reinforcement learning

paper · pdf · doi:10.48550/arxiv.2312.14134

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2023/12/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Learning rewards from expert videos offers an affordable and effective solution to specify the intended behaviors for reinforcement learning (RL) tasks. In this work, we propose Diffusion Reward, a novel framework that learns rewards from expert videos via conditional video diffusion models for solving complex visual RL problems. Our key insight is that lower generative diversity is exhibited when conditioning diffusion on expert trajectories. Diffusion Reward is accordingly formalized by the negative of conditional entropy that encourages productive exploration of expert behaviors. We show the efficacy of our method over robotic manipulation tasks in both simulation platforms and the real world with visual input. Moreover, Diffusion Reward can even solve unseen tasks successfully and effectively, largely surpassing baseline methods. Project page and code: https://diffusion-reward.github.io.

Cited by

• Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representations
• Goal-Driven Reward by Video Diffusion Models for Reinforcement Learning
• GSWorld: Closed-Loop Photo-Realistic Simulation Suite for Robotic Manipulation
• TimeRewarder: Learning Dense Reward from Passive Videos via Frame-wise Temporal Distance
• ReLAM: Learning Anticipation Model for Rewarding Visual Robotic Manipulation
• 4D Visual Pre-training for Robot Learning
• GenFlowRL: Shaping Rewards with Generative Object-Centric Flow in Visual Reinforcement Learning
• Policy Learning from Large Vision-Language Model Feedback without Reward Modeling
• Integrating Diffusion-based Multi-task Learning with Online Reinforcement Learning for Robust Quadruped Robot Control
• AMPLIFY: Actionless Motion Priors for Robot Learning from Videos
• Self-Improving Loops for Visual Robotic Planning
• TeViR: Text-to-Video Reward with Diffusion Models for Efficient Reinforcement Learning
• A Comprehensive Survey on Physical Risk Control in the Era of Foundation Model-enabled Robotics
• Solving New Tasks by Adapting Internet Video Knowledge
• Diffusion Models for Robotic Manipulation: A Survey

Related

• Goal-Driven Reward by Video Diffusion Models for Reinforcement Learning
• Data-regularized Reinforcement Learning for Diffusion Models at Scale
• Extracting Reward Functions from Diffusion Models
• Beyond Human Demonstrations: Diffusion-Based Reinforcement Learning to Generate Data for VLA Training
• TeViR: Text-to-Video Reward with Diffusion Models for Efficient Reinforcement Learning