Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learning

2026/03/12 by Jiaheng Hu, Jay Shim, Chen Tang +4 · 1 voice
Computer Science · #Adaptation (eye) #Domain Adaptation and Few-Shot Learning #Lifelong learning #Multimodal Machine Learning Applications #Recipe #Reinforcement #Reinforcement Learning in Robotics #Reinforcement learning #Robustness (evolution) #Simple (philosophy) #cs.LG #cs.RO

paper · pdf · doi:10.48550/arxiv.2603.11653

openalex publication_date 2026/03/12 · arxiv published 2026/03/12 · openalex created_date 2026/03/14 · arxiv updated 2026/07/11 · openalex updated_date 2026/07/28

Abstract

Continual Reinforcement Learning (CRL) for Vision-Language-Action (VLA) models is a promising direction toward self-improving embodied agents that can adapt in openended, evolving environments. However, conventional wisdom from continual learning suggests that naive Sequential Fine-Tuning (Seq. FT) leads to catastrophic forgetting, necessitating complex CRL strategies. In this work, we take a step back and conduct a systematic study of CRL for large pretrained VLAs across diverse lifelong RL benchmarks. We find that, contrary to established belief, simple Seq. FT with low-rank adaptation (LoRA) is remarkably strong: it achieves high plasticity, exhibits little to no forgetting, and retains strong zero-shot generalization, frequently outperforming more sophisticated CRL methods. Through detailed analysis, we show that this robustness arises from a synergy between the large pretrained model, parameter-efficient adaptation, and on-policy RL. Together, these components reshape the stability-plasticity trade-off, making continual adaptation both stable and scalable. Our results position Sequential Fine-Tuning as a powerful method for continual RL with VLAs and provide new insights into lifelong learning in the large model era. Code is available at https://github.com/UT-Austin-RobIn/continual-vla-rl

Citations

• Pretrained Vision-Language-Action Models are Surprisingly Resistant to Forgetting in Continual Learning
• Robust Finetuning of Vision-Language-Action Robot Policies via Parameter Merging
• π^*_0.6: a VLA That Learns From Experience
• Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting
• RLinf: Flexible and Efficient Large-scale Reinforcement Learning via Macro-to-Micro Flow Transformation
• RL's Razor: Why Online Reinforcement Learning Forgets Less
• Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Training
• A Survey of Continual Reinforcement Learning
• Steering Your Diffusion Policy with Latent Space Reinforcement Learning
• SLAC: Safe and Efficient Real-Robot Reinforcement Learning via Unsupervised Simulation Pre-Training
• Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn
• VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning
• Rethinking the Foundations for Continual Reinforcement Learning
• Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• π₀: A Vision-Language-Action Flow Model for General Robot Control
• FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuning
• OpenVLA: An Open-Source Vision-Language-Action Model
• RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots
• LoRA Learns Less and Forgets Less
• Continual Learning of Large Language Models: A Comprehensive Survey
• Position: Lifetime tuning is incompatible with continual reinforcement learning
• Open X-Embodiment: Robotic Learning Datasets and RT-X Models
• TAIL: Task-specific Adapters for Imitation Learning with Large Pretrained Models
• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
• A Definition of Continual Reinforcement Learning
• LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning
• Loss of Plasticity in Continual Deep Reinforcement Learning
• Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models
• SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model
• Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
• Diffusion policy: Visuomotor policy learning via action diffusion
• ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills
• RT-1: Robotics Transformer for Real-World Control at Scale
• The Effectiveness of World Models for Continual Reinforcement Learning
• On the Effectiveness of Parameter-Efficient Fine-Tuning
• Disentangling Transfer in Continual Reinforcement Learning
• CoMPS: Continual Meta Policy Search
• Wide Neural Networks Forget Less Catastrophically
• Lifelong Robotic Reinforcement Learning by Retaining Experiences
• LoRA: Low-Rank Adaptation of Large Language Models
• Towards Continual Reinforcement Learning: A Review and Perspectives
• The Ingredients of Real-World Robotic Reinforcement Learning
• Dark Experience for General Continual Learning: a Strong, Simple Baseline
• SAPIEN: A SimulAted Part-based Interactive ENvironment
• Policy Consolidation for Continual Reinforcement Learning
• Experience Replay for Continual Learning
• PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning
• Gradient Episodic Memory for Continual Learning
• Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
• Overcoming catastrophic forgetting in neural networks
• Progressive Neural Networks
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Prefix-Tuning: Optimizing Continuous Prompts for Generation

Cited by

• Pretrained Vision-Language-Action Models are Surprisingly Resistant to Forgetting in Continual Learning

Discussions

• 今日読んだ論文 Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learning arxiv.org/abs/2603.11653 On-PolicyRL・VLA・LoRAというシンプルな構成が継続的強化学習において有力であるという説。分析のとこ [bsky, 0 points, 0 comments]

Related

• Pretrained Vision-Language-Action Models are Surprisingly Resistant to Forgetting in Continual Learning
• Can Vision-Language-Action Models Learn from Real-World Data Continually without Forgetting?
• VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulators
• Self-Improving Vision-Language-Action Models with Data Generation via Residual RL
• What Can RL Bring to VLA Generalization? An Empirical Study