ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy

2025/02/08 by Yuhui Chen, Chen, Yuhui, Tian, Shuai +7 · 42 citations
Engineering · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Robotics (cs.RO) #Traffic Prediction and Management Techniques

paper · pdf · doi:10.48550/arxiv.2502.05450

openalex publication_date 2025/02/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Vision-Language-Action (VLA) models have shown substantial potential in real-world robotic manipulation. However, fine-tuning these models through supervised learning struggles to achieve robust performance due to limited, inconsistent demonstrations, especially in contact-rich environments. In this paper, we propose a reinforced fine-tuning approach for VLA models, named ConRFT, which consists of offline and online fine-tuning with a unified consistency-based training objective, to address these challenges. In the offline stage, our method integrates behavior cloning and Q-learning to effectively extract policy from a small set of demonstrations and stabilize value estimating. In the online stage, the VLA model is further fine-tuned via consistency policy, with human interventions to ensure safe exploration and high sample efficiency. We evaluate our approach on eight diverse real-world manipulation tasks. It achieves an average success rate of 96.3% within 45-90 minutes of online fine-tuning, outperforming prior supervised methods with a 144% improvement in success rate and 1.9x shorter episode length. This work highlights the potential of integrating reinforcement learning to enhance the performance of VLA models for real-world robotic applications. Videos and code are available at our project website https://cccedric.github.io/conrft/.

Cited by

• Embodied Robot Manipulation in the Era of Foundation Models: Planning and Learning Perspectives
• Robo-Dopamine: General Process Reward Modeling for High-Precision Robotic Manipulation
• Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representations
• Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuning
• An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challenges
• STARE-VLA: Progressive Stage-Aware Reinforcement for Fine-Tuning Vision-Language-Action Models
• Beyond Success: Refining Elegant Robot Manipulation from Mixed-Quality Data via Just-in-Time Intervention
• Discover, Learn, and Reinforce: Scaling Vision-Language-Action Pretraining with Diverse RL-Generated Trajectories
• DiffuDepGrasp: Diffusion-based Depth Noise Modeling Empowers Sim2Real Robotic Grasping
• π^*_0.6: a VLA That Learns From Experience
• RobustVLA: Robustness-Aware Reinforcement Post-Training for Vision-Language-Action Models
• OpenVLN: Open-world Aerial Vision-Language Navigation
• DeepThinkVLA: Enhancing Reasoning Capability of Vision-Language-Action Models
• Self-Improving Vision-Language-Action Models with Data Generation via Residual RL
• A Survey on Efficient Vision-Language-Action Models
• UrbanVLA: A Vision-Language-Action Model for Urban Micromobility
• RESample: A Robust Data Augmentation Framework via Exploratory Sampling for Robotic Manipulation
• Efficient Vision-Language-Action Models for Embodied Manipulation: A Systematic Survey
• VLA²: Empowering Vision-Language-Action Models with an Agentic Framework for Unseen Concept Manipulation
• QDepth-VLA: Quantized Depth Prediction as Auxiliary Supervision for Vision-Language-Action Models
• Reflection-Based Task Adaptation for Self-Improving VLA
• DEAS: DEtached value learning with Action Sequence for Scalable Offline RL
• Vision-Language-Action Models for Robotics: A Review Towards Real-World Applications
• Self-Improvement in Multimodal Large Language Models: A Survey
• VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulators
• VLA Model Post-Training via Action-Chunked PPO and Self Behavior Cloning
• PhysiAgent: An Embodied Agent Framework in Physical World
• Teaching RL Agents to Act Better: VLM as Action Advisor for Online Reinforcement Learning
• RL²-VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models
• SOE: Sample-Efficient Robot Policy Self-Improvement via On-Manifold Exploration
• Pure Vision Language Action (VLA) Models: A Comprehensive Survey
• RoboSeek: You Need to Interact with Your Objects
• A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learning
• ExT: Towards Scalable Autonomous Excavation via Large-Scale Multi-Task Pretraining and Fine-Tuning
• Dual-Actor Fine-Tuning of VLA Models: A Talk-and-Tweak Human-in-the-Loop Approach
• SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning
• Survey of Vision-Language-Action Models for Embodied Manipulation
• Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulation
• Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Survey
• Reinforcement Learning for Large Model: A Survey
• DiWA: Diffusion Policy Adaptation with World Models
• CO-RFT: Efficient Fine-Tuning of Vision-Language-Action Models through Chunked Offline Reinforcement Learning

Related

• CO-RFT: Efficient Fine-Tuning of Vision-Language-Action Models through Chunked Offline Reinforcement Learning
• Dual-Actor Fine-Tuning of VLA Models: A Talk-and-Tweak Human-in-the-Loop Approach
• VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning
• Improving Vision-Language-Action Model with Online Reinforcement Learning
• SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning