Reinforcement Learning with Action Chunking

2025/07/10 by Qiyang Li, Zhiyuan Zhou, Li, Qiyang +3 · 23 citations
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.2507.07969

openalex publication_date 2025/07/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present Q-chunking, a simple yet effective recipe for improving reinforcement learning (RL) algorithms for long-horizon, sparse-reward tasks. Our recipe is designed for the offline-to-online RL setting, where the goal is to leverage an offline prior dataset to maximize the sample-efficiency of online learning. Effective exploration and sample-efficient learning remain central challenges in this setting, as it is not obvious how the offline data should be utilized to acquire a good exploratory policy. Our key insight is that action chunking, a technique popularized in imitation learning where sequences of future actions are predicted rather than a single action at each timestep, can be applied to temporal difference (TD)-based RL methods to mitigate the exploration challenge. Q-chunking adopts action chunking by directly running RL in a 'chunked' action space, enabling the agent to (1) leverage temporally consistent behaviors from offline data for more effective online exploration and (2) use unbiased n-step backups for more stable and efficient TD learning. Our experimental results demonstrate that Q-chunking exhibits strong offline performance and online sample efficiency, outperforming prior best offline-to-online methods on a range of long-horizon, sparse-reward manipulation tasks.

Citations

• Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returns
• Flow Q-Learning
• Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data
• OGBench: Benchmarking Offline Goal-Conditioned RL
• Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration
• Diffusion Policy Policy Optimization
• Continuous Control with Coarse-to-fine Reinforcement Learning
• Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodings
• Foundation Policies with Hilbert Representations
• Accelerating Exploration with Unlabeled Prior Data
• Unsupervised Behavior Extraction via Random Intent Priors
• Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learning
• One ACT Play: Single Demonstration Behavior Cloning with Action Chunking Transformers
• RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunking
• Self-Supervised Reinforcement Learning that Transfers using Random Features
• Revisiting the Minimalist Approach to Offline Reinforcement Learning
• Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware
• Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
• Adaptive Policy Learning for Offline-to-Online Reinforcement Learning
• Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
• Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
• Diffusion policy: Visuomotor policy learning via action diffusion
• Efficient Online Reinforcement Learning with Offline Data
• Policy Expansion for Bridging Offline-to-Online Reinforcement Learning
• Learning and Retrieval from Prior Data for Skill-based Imitation Learning
• Hybrid RL: Using Both Offline and Online Data Can Make RL Efficient
• Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
• Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progress
• Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives
• Hierarchical Skills for Efficient Exploration
• Offline Reinforcement Learning with Implicit Q-Learning
• What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
• Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble
• Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning
• Revisiting Peng's Q(λ) for Modern Reinforcement Learning
• Latent Skill Planning for Exploration and Transfer
• Parrot: Data-Driven Behavioral Priors for Reinforcement Learning
• OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning
• Accelerating Reinforcement Learning with Learned Skill Priors
• Learning to summarize from human feedback
• EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL
• Revisiting Fundamentals of Experience Replay
• Learning Robot Skills with Temporal Variational Inference
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• Conservative Q-Learning for Offline Reinforcement Learning
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
• Variational Temporal Abstraction
• Neural probabilistic motor primitives for humanoid control
• Data-Efficient Hierarchical Reinforcement Learning
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Value Prediction Network
• Multi-Level Discovery of Deep Options
• FeUdal Networks for Hierarchical Reinforcement Learning
• The Option-Critic Architecture
• Strategic Attentive Writer for Learning Macro-Actions
• Deep Reinforcement Learning With Macro-Actions
• Option Discovery in Hierarchical Reinforcement Learning using Spatio-Temporal Clustering
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• Asynchronous Methods for Deep Reinforcement Learning
• Continuous control with deep reinforcement learning
• Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
• Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition
• Coarse-to-fine Q-Network with Action Sequence for Data-Efficient Reinforcement Learning

Cited by

• Action-Sufficient Goal Representations
• Scalable Offline Model-Based RL with Action Chunks
• GR-RL: Going Dexterous and Precise for Long-Horizon Robotic Manipulation
• Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learning
• Multi-agent Coordination via Flow Matching
• Temporal Action Selection for Action Chunking
• From Static Constraints to Dynamic Adaptation: Sample-Level Constraint Relaxation for Offline-to-Online Reinforcement Learning
• Human-in-the-loop Online Rejection Sampling for Robotic Manipulation
• Self-Improving Vision-Language-Action Models with Data Generation via Residual RL
• Sample By Step, Optimize By Chunk: Chunk-Level GRPO For Text-to-Image Generation
• Expressive Value Learning for Scalable Offline Reinforcement Learning
• DEAS: DEtached value learning with Action Sequence for Scalable Offline RL
• Value Flows
• SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modeling
• In-Context Compositional Q-Learning for Offline Reinforcement Learning
• SOE: Sample-Efficient Robot Policy Self-Improvement via On-Manifold Exploration
• FinFlowRL: An Imitation-Reinforcement Learning Framework for Adaptive Stochastic Control in Finance
• Ratatouille: Imitation Learning Ingredients for Real-world Social Robot Navigation
• A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learning
• Empowering Multi-Robot Cooperation via Sequential World Models
• FinFlowRL: An Imitation-Reinforcement Learning Framework for Adaptive Stochastic Control in Finance
• ReWiND: Language-Guided Rewards Teach Robot Policies without New Demonstrations
• Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities
• EvoHIL: Self-Evolving Reward and Flow-Matched Policy Optimization for Robust Human-in-the-Loop Reinforcement Learning

Related

• SEAR: Sample Efficient Action Chunking Reinforcement Learning
• Fill in the Blanks: Accelerating Q-Learning with a Handful of Demonstrations in Sparse Reward Settings
• Scalable Offline Model-Based RL with Action Chunks
• Offline RL with Hierarchical Action Chunking
• Reasoning with Latent Diffusion in Offline Reinforcement Learning