floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RL

2025/09/08 by Bhavya Agrawalla, Michal Nauman, Agrawalla, Bhavya +4 · 1 citation
Business, Management and Accounting · Computer Science · #Artificial Intelligence (cs.AI) #Business Process Modeling and Analysis #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multi-Agent Systems and Negotiation #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2509.06863

openalex publication_date 2025/09/08 · openalex created_date 2025/10/11 · openalex updated_date 2026/07/28

Abstract

A hallmark of modern large-scale machine learning techniques is the use of training objectives that provide dense supervision to intermediate computations, such as teacher forcing the next token in language models or denoising step-by-step in diffusion models. This enables models to learn complex functions in a generalizable manner. Motivated by this observation, we investigate the benefits of iterative computation for temporal difference (TD) methods in reinforcement learning (RL). Typically they represent value functions in a monolithic fashion, without iterative compute. We introduce floq (flow-matching Q-functions), an approach that parameterizes the Q-function using a velocity field and trains it using techniques from flow-matching, typically used in generative modeling. This velocity field underneath the flow is trained using a TD-learning objective, which bootstraps from values produced by a target velocity field, computed by running multiple steps of numerical integration. Crucially, floq allows for more fine-grained control and scaling of the Q-function capacity than monolithic architectures, by appropriately setting the number of integration steps. Across a suite of challenging offline RL benchmarks and online fine-tuning tasks, floq improves performance by nearly 1.8x. floq scales capacity far better than standard TD-learning architectures, highlighting the potential of iterative computation for value learning.

Citations

• Compute-Optimal Scaling for Value-Based Deep RL
• Steering Your Diffusion Policy with Latent Space Reinforcement Learning
• Horizon Reduction Makes RL Scalable
• Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners
• FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Control
• Scaling Offline RL via Efficient and Expressive Shortcut Models
• 1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities
• Temporal Difference Flows
• Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs
• Hyperspherical Normalization for Scalable Deep Reinforcement Learning
• Value-Based Deep RL Scales Predictably
• Flow Q-Learning
• Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone
• OGBench: Benchmarking Offline Goal-Conditioned RL
• The Ingredients for Robotic Diffusion Transformers
• SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning
• Diffusion Policy Policy Optimization
• D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learning
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
• Is Value Learning Really the Main Bottleneck in Offline RL?
• Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradient
• Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control
• Stop Regressing: Training Value Functions via Classification for Scalable Deep RL
• Mixtures of Experts Unlock Parameter Scaling for Deep RL
• Offline Actor-Critic Reinforcement Learning Scales to Large Models
• TD-MPC2: Scalable, Robust World Models for Continuous Control
• Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions
• Policy Representation via Diffusion Probability Model for Reinforcement Learning
• Revisiting the Minimalist Approach to Offline Reinforcement Learning
• IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies
• Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization
• Extreme Q-Learning: MaxEnt RL without Entropy
• Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes
• Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trials
• Flow Matching for Generative Modeling
• Building Normalizing Flows with Stochastic Interpolants
• Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL
• Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
• Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise
• Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning
• An Empirical Study of Implicit Regularization in Deep Offline RL
• Contrastive Learning as Goal-Conditioned Reinforcement Learning
• Multi-Game Decision Transformers
• Planning with Diffusion for Flexible Behavior Synthesis
• Temporal Difference Learning for Model Predictive Control
• DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization
• Mastering Atari Games with Limited Data
• Offline Reinforcement Learning with Implicit Q-Learning
• Deep Reinforcement Learning at the Edge of the Statistical Precipice
• A Minimalist Approach to Offline Reinforcement Learning
• Offline Reinforcement Learning as One Big Sequence Modeling Problem
• Towards Deeper Deep Reinforcement Learning with Spectral Normalization
• Learning and Planning in Complex Action Spaces
• On The Effect of Auxiliary Tasks on Representation Dynamics
• Score-Based Generative Modeling through Stochastic Differential Equations
• Implicit Under-Parameterization Inhibits Data-Efficient Deep\n Reinforcement Learning
• Generative Temporal Difference Learning for Infinite-Horizon Prediction
• Model-Based Offline Planning
• Critic Regularized Regression
• Denoising Diffusion Probabilistic Models
• Conservative Q-Learning for Offline Reinforcement Learning
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Mastering Atari, Go, chess and shogi by planning with a learned model
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• Behavior Regularized Offline Reinforcement Learning
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity
• Soft Actor-Critic Algorithms and Applications
• Off-Policy Deep Reinforcement Learning without Exploration
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning
• Distributional Reinforcement Learning with Quantile Regression
• Attention Is All You Need
• Gaussian Error Linear Units (GELUs)
• Deep Residual Learning for Image Recognition
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics

Cited by

• RAMAC: Multimodal Risk-Aware Offline Reinforcement Learning and the Role of Behavior Regularization

Related

• Q-VGM: Q-Value-Gradient Matching for Off-Policy Reinforcement Learning of Flow-Matching VLA
• Flow Q-Learning
• Flow-Based Policy for Online Reinforcement Learning
• One-Step Flow Q-Learning: Addressing the Diffusion Policy Bottleneck in Offline Reinforcement Learning
• JustRL: Scaling a 1.5B LLM with a Simple RL Recipe