Critic Regularized Regression

2020/06/26 by Ziyu Wang, Alexander Novikov, Wang, Ziyu +19 · 12 citations
Computer Science · #Adaptive Dynamic Programming Control #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2006.15134

openalex publication_date 2020/06/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Offline reinforcement learning (RL), also known as batch RL, offers the prospect of policy optimization from large pre-recorded datasets without online environment interaction. It addresses challenges with regard to the cost of data collection and safety, both of which are particularly pertinent to real-world applications of RL. Unfortunately, most off-policy algorithms perform poorly when learning from a fixed dataset. In this paper, we propose a novel offline RL algorithm to learn policies from data using a form of critic-regularized regression (CRR). We find that CRR performs surprisingly well and scales to tasks with high-dimensional state and action spaces -- outperforming several state-of-the-art offline RL algorithms by a significant margin on a wide range of benchmark tasks.

Citations

• Adam: A Method for Stochastic Optimization
• DeepMind Control Suite
• Challenges of Real-World Reinforcement Learning
• A Reduction of Imitation Learning and Structured Prediction to No-Regret\n Online Learning
• Benchmarking Batch Deep Reinforcement Learning Algorithms
• BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement\n Learning
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems

Cited by

• Goal Reaching with Eikonal-Constrained Hierarchical Quasimetric Reinforcement Learning
• Real-World Reinforcement Learning of Active Perception Behaviors
• π^*_0.6: a VLA That Learns From Experience
• Online and Offline Reinforcement Learning by Planning with a Learned Model
• Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learning
• Beyond Static LLM Policies: Imitation-Enhanced Reinforcement Learning for Recommendation
• Expert or not? assessing data quality in offline reinforcement learning
• DEAS: DEtached value learning with Action Sequence for Scalable Offline RL
• A KL-regularization Framework for Learning to Plan with Adaptive Priors
• floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RL
• Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learning
• Offline Reinforcement Learning with Soft Behavior Regularization

Related

• Offline RL Without Off-Policy Evaluation
• Iteratively Refined Behavior Regularization for Offline Reinforcement Learning
• BRAC+: Improved Behavior Regularized Actor Critic for Offline Reinforcement Learning
• A Connection between One-Step Regularization and Critic Regularization in Reinforcement Learning
• Behavior Regularized Offline Reinforcement Learning