Scaling Offline RL via Efficient and Expressive Shortcut Models

2025/05/28 by Nicolas Espinosa-Dice, Espinosa-Dice, Nicolas, Yiyi Zhang +16 · 1 voice · 3 citations
Computer Science · #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #I.2.6 #Machine Learning (cs.LG) #Machine Learning and Data Classification #Reinforcement Learning in Robotics #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.2505.22866

openalex publication_date 2025/05/28 · arxiv published 2025/05/28 · arxiv updated 2025/05/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Diffusion and flow models have emerged as powerful generative approaches capable of modeling diverse and multimodal behavior. However, applying these models to offline reinforcement learning (RL) remains challenging due to the iterative nature of their noise sampling processes, making policy optimization difficult. In this paper, we introduce Scalable Offline Reinforcement Learning (SORL), a new offline RL algorithm that leverages shortcut models - a novel class of generative models - to scale both training and inference. SORL's policy can capture complex data distributions and can be trained simply and efficiently in a one-stage training procedure. At test time, SORL introduces both sequential and parallel inference scaling by using the learned Q-function as a verifier. We demonstrate that SORL achieves strong performance across a range of offline RL tasks and exhibits positive scaling behavior with increased test-time compute. We release the code at nico-espinosadice.github.io/projects/sorl.

Citations

• Learning Adaptive Parallel Reasoning with Language Models
• To Backtrack or Not to Backtrack: When Sequential Search Limits Model Reasoning
• AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems
• Energy-Weighted Flow Matching for Offline Reinforcement Learning
• All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning
• Flow Q-Learning
• s1: Simple test-time scaling
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Steps
• A General Framework for Inference-time Scaling and Steering of Diffusion Models
• Flow Matching Guide and Code
• π₀: A Vision-Language-Action Flow Model for General Robot Control
• OGBench: Benchmarking Offline Goal-Conditioned RL
• Diffusing States and Matching Scores: A New Framework for Imitation Learning
• One Step Diffusion via Shortcut Models
• Training Language Models to Self-Correct via Reinforcement Learning
• Diffusion Policy Policy Optimization
• D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learning
• Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
• Recursive Introspection: Teaching Language Model Agents How to Self-Improve
• Is Value Learning Really the Main Bottleneck in Offline RL?
• BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling
• Diffusion Policies creating a Trust Region for Offline Reinforcement Learning
• Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
• Hybrid Inverse Reinforcement Learning
• Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning
• Improved Techniques for Training Consistency Models
• Score Regularized Policy Optimization through Diffusion Behavior
• DiffCPS: Diffusion Model based Constrained Policy Search for Offline Reinforcement Learning
• Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning
• Benchmarking Offline Reinforcement Learning on Real-Robot Hardware
• Robotic Manipulation Datasets for Offline Compositional Reinforcement Learning
• Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning
• Let's Verify Step by Step
• Efficient Diffusion Policies for Offline Reinforcement Learning
• Revisiting the Minimalist Approach to Offline Reinforcement Learning
• IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies
• Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization
• Stochastic Interpolants: A Unifying Framework for Flows and Diffusions
• Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
• Dual RL: Unification and New Methods for Reinforcement and Imitation Learning
• Efficient Online Reinforcement Learning with Offline Data
• Anti-Exploration by Random Network Distillation
• Extreme Q-Learning: MaxEnt RL without Entropy
• CORL: Research-oriented Deep Offline Reinforcement Learning Library
• Hybrid RL: Using Both Offline and Online Data Can Make RL Efficient
• On Distillation of Guided Diffusion Models
• Flow Matching for Generative Modeling
• Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling
• Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
• Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning
• Planning with Diffusion for Flexible Behavior Synthesis
• Self-Consistency Improves Chain of Thought Reasoning in Language Models
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Offline Reinforcement Learning with Implicit Q-Learning
• Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble
• OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation
• A Minimalist Approach to Offline Reinforcement Learning
• Offline Reinforcement Learning as One Big Sequence Modeling Problem
• Decision Transformer: Reinforcement Learning via Sequence Modeling
• Score-Based Generative Modeling through Stochastic Differential Equations
• Denoising Diffusion Implicit Models
• EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL
• Denoising Diffusion Probabilistic Models
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• Conservative Q-Learning for Offline Reinforcement Learning
• MOPO: Model-based Offline Policy Optimization
• MOReL : Model-Based Offline Reinforcement Learning
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Behavior Regularized Offline Reinforcement Learning
• Off-Policy Deep Reinforcement Learning without Exploration
• Layer Normalization
• Gaussian Error Linear Units (GELUs)
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• Adam: A Method for Stochastic Optimization
• Playing Atari with Deep Reinforcement Learning

Cited by

• One-Step Generative Policies with Q-Learning: A Reformulation of MeanFlow
• Expressive Value Learning for Scalable Offline Reinforcement Learning
• floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RL

Discussions

• Scalable Offline RL (𝚂𝙾𝚁𝙻): a training procedure for *expressive policies* that can do inference under *any* compute budget paper: arxiv.org/abs/2505.22866 code: github.com/nico-espinosadice/SORL [bsky, 0 points, 1 comments]

Related

• Horizon Reduction Makes RL Scalable
• Expressive Value Learning for Scalable Offline Reinforcement Learning
• A Minimalist Approach to Offline Reinforcement Learning
• Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes
• A Tractable Inference Perspective of Offline RL