RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback

2024/02/06 by Yufei Wang, Wang, Yufei, Zhanyi Sun +11 · 28 citations
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Fuzzy Logic and Control Systems #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Robotics (cs.RO) #Robotics and Automated Systems

paper · pdf · doi:10.48550/arxiv.2402.03681

openalex publication_date 2024/02/06 · openalex created_date 2024/02/08 · openalex updated_date 2026/07/28

Abstract

Reward engineering has long been a challenge in Reinforcement Learning (RL) research, as it often requires extensive human effort and iterative processes of trial-and-error to design effective reward functions. In this paper, we propose RL-VLM-F, a method that automatically generates reward functions for agents to learn new tasks, using only a text description of the task goal and the agent's visual observations, by leveraging feedbacks from vision language foundation models (VLMs). The key to our approach is to query these models to give preferences over pairs of the agent's image observations based on the text description of the task goal, and then learn a reward function from the preference labels, rather than directly prompting these models to output a raw reward score, which can be noisy and inconsistent. We demonstrate that RL-VLM-F successfully produces effective rewards and policies across various domains - including classic control, as well as manipulation of rigid, articulated, and deformable objects - without the need for human supervision, outperforming prior methods that use large pretrained models for reward generation under the same assumptions. Videos can be found on our project website: https://rlvlmf2024.github.io/

Cited by

• ReasonX: MLLM-Guided Intrinsic Image Decomposition
• An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challenges
• SIMA 2: A Generalist Embodied Agent for Virtual Worlds
• BiTAgent: A Task-Aware Modular Framework for Bidirectional Coupling between Multimodal Large Language Models and World Models
• Leveraging LLMs for reward function design in reinforcement learning control tasks
• AutoFocus-IL: VLM-based Saliency Maps for Data-Efficient Visual Imitation Learning without Extra Human Annotations
• PROF: An LLM-based Reward Code Preference Optimization Framework for Offline Imitation Learning
• GenDexHand: Generative Simulation for Dexterous Hands
• World-in-World: World Models in a Closed-Loop World
• Talking Points: Describing and Localizing Pixels
• Reflection-Based Task Adaptation for Self-Improving VLA
• CDE: Concept-Driven Exploration for Reinforcement Learning
• Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play
• Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curation
• LAGEA: Language Guided Embodied Agents for Robotic Manipulation
• What If Moderation Didn't Mean Suppression? A Case for Personalized Content Transformation
• Teaching RL Agents to Act Better: VLM as Action Advisor for Online Reinforcement Learning
• A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learning
• How well can LLMs provide planning feedback in grounded environments?
• Aligning Large Vision-Language Models by Deep Reinforcement Learning and Direct Preference Optimization
• Language-Guided Long Horizon Manipulation with LLM-based Planning and Visual Perception
• In-Context Iterative Policy Improvement for Dynamic Manipulation
• UAV-VL-R1: Generalizing Vision-Language Models via Supervised Fine-Tuning and Multi-Stage GRPO for UAV Visual Reasoning
• Pref-GUIDE: Continual Policy Learning from Real-Time Human Feedback via Preference-Based Learning
• ROVER: Recursive Reasoning Over Videos with Vision-Language Models for Embodied Tasks
• HALO: Human Preference Aligned Offline Reward Learning for Robot Navigation
• Policy Learning from Large Vision-Language Model Feedback without Reward Modeling
• Bridging the Gap in Vision Language Models in Identifying Unsafe Concepts Across Modalities

Related

• Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning
• Code as Reward: Empowering Reinforcement Learning with VLMs
• Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models
• VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model
• Reinforcement Learning for Large Model: A Survey