RL²-VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models

2026/07/30 by Derek Ming Siang Tan, Shailesh Shailesh, Srikrishna Iyer +4
Computer Science · #cs.RO

paper · pdf

Code and models are available at https://rl2-vla.github.io

arxiv created 2026/07/30 · arxiv updated 2026/07/31

Abstract

Despite the impressive visuomotor capabilities enabled by Vision-Language-Action (VLA) models, their performance often degrades on challenging and out-of-domain tasks. Recent test-time steering and scaling methods improve performance without extensive data collection and retraining, but action samples often remain concentrated around similar behaviors and therefore inherit correlated failure modes. Moreover, existing methods apply the same intervention strategy at every timestep, regardless of whether the base policy is already likely to succeed. To address these limitations, we introduce RL², an adaptive inference-time steering framework that leverages Reinforcement Learning on VLA Latents. First, we train a lightweight offline RL policy conditioned on expressive latents extracted from the VLA action expert and compose its flow velocity with that of the frozen VLA during inference. This compositional steering strategy combines the behavioral priors of large-scale imitation learning with the action diversity induced by offline RL beyond dominant demonstration modes. We further discover that inference-time steering follows fundamentally different scaling laws under success and failure states, revealing that action diversity is most beneficial when the base VLA is likely to fail, but can unnecessarily perturb already-accurate actions when success is likely. Building on this insight, RL² activates compositional steering only when failure is predicted. Across the SIMPLER and PolaRiS benchmarks, RL² improves success rates by up to +17.3% in out-of-domain settings, while ablations and scaling studies demonstrate the importance of latent representations and RL training. Finally, real-world experiments demonstrate that these gains transfer beyond simulation, establishing RL² as a practical and modular steering framework for VLA deployment.

Citations

• Towards Deploying VLA without Fine-Tuning: Plug-and-Play Inference-Time VLA Policy Steering via Embodied Evolutionary Diffusion
• π^*_0.6: a VLA That Learns From Experience
• Self-Improving Vision-Language-Action Models with Data Generation via Residual RL
• Do What You Say: Steering Vision-Language-Action Models via Runtime Reasoning-Action Alignment Verification
• Compose Your Policies! Improving Diffusion-based or Flow-based Robot Policies via Test-time Distribution-level Composition
• SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning
• Tree-Guided Diffusion Planner
• MolmoAct: Action Reasoning Models that can Reason in Space
• RoboMonkey: Scaling Test-Time Sampling and Verification for Vision-Language-Action Models
• Steering Your Diffusion Policy with Latent Space Reinforcement Learning
• DynaGuide: Steering Diffusion Polices with Active Dynamic Guidance
• From Intention to Execution: Probing the Generalization Boundaries of Vision-Language-Action Models
• SAFE: Multitask Failure Detection for Vision-Language-Action Models
• Fast ECoT: Efficient Embodied Chain-of-Thought via Thoughts Reuse
• OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoning
• π_0.5: a Vision-Language-Action Model with Open-World Generalization
• Can We Detect Failures Without Failure Data? Uncertainty-Aware Runtime Failure Detection for Imitation Learning Policies
• ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy
• Flow Q-Learning
• From Foresight to Forethought: VLM-In-the-Loop Policy Steering via Latent Alignment
• Policy Decorator: Model-Agnostic Online Refinement for Large Policy Model
• GRAPE: Generalizing Robot Policy via Preference Alignment
• Embodied Red Teaming for Auditing Robotic Foundation Models
• π₀: A Vision-Language-Action Flow Model for General Robot Control
• OGBench: Benchmarking Offline Goal-Conditioned RL
• Steering Your Generalists: Improving Robotic Foundation Models via Value Guidance
• Robotic Control via Embodied Chain-of-Thought Reasoning
• OpenVLA: An Open-Source Vision-Language-Action Model
• Evaluating Real-World Robot Manipulation Policies in Simulation
• DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
• BridgeData V2: A Dataset for Robot Learning at Scale
• Diffusion Models Beat GANs on Image Synthesis
• Conservative Q-Learning for Offline Reinforcement Learning

Related

• Steering Vision-Language-Action Models as Anti-Exploration: A Test-Time Scaling Approach
• Self-Improving Vision-Language-Action Models with Data Generation via Residual RL
• What Can RL Bring to VLA Generalization? An Empirical Study
• VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulators
• Discover, Learn, and Reinforce: Scaling Vision-Language-Action Pretraining with Diverse RL-Generated Trajectories