Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions

2023/09/18 by Yevgen Chebotar, Chebotar, Yevgen, Quan Vuong +48 · 1 voice · 35 citations
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #Robot Manipulation and Learning #Robotics (cs.RO) #cs.AI #cs.LG #cs.RO

paper · pdf · doi:10.48550/arxiv.2309.10150

openalex publication_date 2023/09/18 · arxiv published 2023/09/18 · arxiv updated 2023/10/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this work, we present a scalable reinforcement learning method for training multi-task policies from large offline datasets that can leverage both human demonstrations and autonomously collected data. Our method uses a Transformer to provide a scalable representation for Q-functions trained via offline temporal difference backups. We therefore refer to the method as Q-Transformer. By discretizing each action dimension and representing the Q-value of each action dimension as separate tokens, we can apply effective high-capacity sequence modeling techniques for Q-learning. We present several design decisions that enable good performance with offline RL training, and show that Q-Transformer outperforms prior offline RL algorithms and imitation learning techniques on a large diverse real-world robotic manipulation task suite. The project's website and videos can be found at https://qtransformer.github.io

Cited by

• Masked Generative Policy for Robotic Control
• Natural Language Actor-Critic: Scalable Off-Policy Learning in Language Space
• RobustVLA: Robustness-Aware Reinforcement Post-Training for Vision-Language-Action Models
• Gentle Manipulation Policy Learning via Demonstrations from VLM Planned Atomic Skills
• OmniVLA: Physically-Grounded Multimodal VLA with Unified Multi-Sensor Perception for Robotic Manipulation
• GauDP: Reinventing Multi-Agent Collaboration through Gaussian-Image Synergy in Diffusion Policies
• Human-in-the-loop Online Rejection Sampling for Robotic Manipulation
• NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigation
• Sequential Multi-Agent Dynamic Algorithm Configuration
• Utilizing Model-Free Reinforcement Learning for Optimizing Secure Multi-Party Computation Protocols
• DEAS: DEtached value learning with Action Sequence for Scalable Offline RL
• ELMUR: External Layer Memory with Update/Rewrite for Long-Horizon RL Problems
• Accelerating Transformers in Online RL
• DipLLM: Fine-Tuning LLM for Strategic Decision-making in Diplomacy
• SeedPolicy: Horizon Scaling via Self-Evolving Diffusion Policy for Robot Manipulation
• floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RL
• Hybrid Sequence Modeling and Reinforced Verification for Controllable Target-Conditioned Decision Making
• Optimal Dynamic Regret by Transformers for Non-Stationary Reinforcement Learning
• TrimTokenator: Towards Adaptive Visual Token Pruning for Large Multimodal Models
• Large Model Empowered Embodied AI: A Survey on Decision-Making and Embodied Learning
• Should We Ever Prefer Decision Transformer for Offline Reinforcement Learning?
• A Survey on Imitation Learning for Contact-Rich Tasks in Robotics
• Horizon Reduction Makes RL Scalable
• Normalizing Flows are Capable Models for Continuous Control
• BOFormer: Learning to Solve Multi-Objective Bayesian Optimization via Non-Markovian RL
• Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RL
• Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skills
• SAINT: Attention-Based Policies for Discrete Combinatorial Action Spaces
• Hierarchical Surgical Robot Transformer (SRT-H): Imitation Learning for Autonomous Surgery
• ReinboT: Amplifying Robot Visual-Language Manipulation with Reinforcement Learning
• Pretraining a Shared Q-Network for Data-Efficient Offline Reinforcement Learning
• Self-Correcting VLA: Online Action Refinement via Sparse World Imagination
• Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities
• ValueFormer: A Causal Transformer Value Function with Stage-Aware Labels for Semi-Autonomous Vision-Language-Action Policies
• RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins

Discussions

• 📝 Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions 🦾🧠 "Q-Transformer leverages the Transformer architecture, a state-of-the-art architecture for sequence modeli [mastodon, 0 points, 0 comments]

Related

• Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL
• Stabilizing Transformer-Based Action Sequence Generation For Q-Learning
• Q-value Regularized Transformer for Offline Reinforcement Learning
• Composite Q-learning: Multi-scale Q-function Decomposition and Separable Optimization
• S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning