Controllable Flow Matching for Online Reinforcement Learning

2025/11/10 by Bin Wang, Wang, Bin, Haifeng Jing +5
Computer Science · Engineering · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #Robot Manipulation and Learning

paper · pdf · doi:10.48550/arxiv.2511.06816

openalex publication_date 2025/11/10 · openalex created_date 2025/11/12 · openalex updated_date 2026/07/28

Abstract

Model-based reinforcement learning (MBRL) typically relies on modeling environment dynamics for data efficiency. However, due to the accumulation of model errors over long-horizon rollouts, such methods often face challenges in maintaining modeling stability. To address this, we propose CtrlFlow, a trajectory-level synthetic method using conditional flow matching (CFM), which directly modeling the distribution of trajectories from initial states to high-return terminal states without explicitly modeling the environment transition function. Our method ensures optimal trajectory sampling by minimizing the control energy governed by the non-linear Controllability Gramian Matrix, while the generated diverse trajectory data significantly enhances the robustness and cross-task generalization of policy learning. In online settings, CtrlFlow demonstrates the better performance on common MuJoCo benchmark tasks than dynamics models and achieves superior sample efficiency compared to standard MBRL methods.

Citations

• Control analysis and synthesis for general control-affine systems
• Learning Transformer-based World Models with Contrastive Predictive Coding
• On the Guidance of Flow Matching
• Consistency Flow Matching: Defining Straight Flows with Velocity Consistency
• Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning
• Diffusion Actor-Critic with Entropy Regulator
• Model-Based Control with Sparse Neural Dynamics
• Learning to Reach Goals via Diffusion
• Informed POMDP: Leveraging Additional Information in Model-Based RL
• AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners
• Building Normalizing Flows with Stochastic Interpolants
• Model-based Reinforcement Learning with Multi-step Plan Value Estimation
• Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
• Planning with Diffusion for Flexible Behavior Synthesis
• When to Trust Your Model: Model-Based Policy Optimization
• Combating the Compounding-Error Problem with a Multi-step Model
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• Model-Ensemble Trust-Region Policy Optimization
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Attention Is All You Need
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics

Related

• Composite Flow Matching for Reinforcement Learning with Shifted-Dynamics Data
• FM-IRL: Flow-Matching for Reward Modeling and Policy Regularization in Reinforcement Learning
• Flow-Based Policy for Online Reinforcement Learning
• Model-based deep reinforcement learning for accelerated learning from flow simulations
• SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modeling