OGBench: Benchmarking Offline Goal-Conditioned RL

2024/10/26 by Seohong Park, Kevin Frans, Park, Seohong +5 · 39 citations
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Model-Driven Software Engineering Techniques

paper · pdf · doi:10.48550/arxiv.2410.20092

openalex publication_date 2024/10/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Offline goal-conditioned reinforcement learning (GCRL) is a major problem in reinforcement learning (RL) because it provides a simple, unsupervised, and domain-agnostic way to acquire diverse behaviors and representations from unlabeled data without rewards. Despite the importance of this setting, we lack a standard benchmark that can systematically evaluate the capabilities of offline GCRL algorithms. In this work, we propose OGBench, a new, high-quality benchmark for algorithms research in offline goal-conditioned RL. OGBench consists of 8 types of environments, 85 datasets, and reference implementations of 6 representative offline GCRL algorithms. We have designed these challenging and realistic environments and datasets to directly probe different capabilities of algorithms, such as stitching, long-horizon reasoning, and the ability to handle high-dimensional inputs and stochasticity. While representative algorithms may rank similarly on prior benchmarks, our experiments reveal stark strengths and weaknesses in these different capabilities, providing a strong foundation for building new algorithms. Project page: https://seohong.me/projects/ogbench

Cited by

• Action from Adjacent Set in Physical Space Outperforms the Best Prediction in World Models
• Action-Sufficient Goal Representations
• Planning as Descent: Goal-Conditioned Latent Trajectory Synthesis in Learned Energy Landscapes
• Goal Reaching with Eikonal-Constrained Hierarchical Quasimetric Reinforcement Learning
• Scalable Offline Model-Based RL with Action Chunks
• Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learning
• Forecasting in Offline Reinforcement Learning for Non-stationary Environments
• Real-World Reinforcement Learning of Active Perception Behaviors
• SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
• ASTRO: Adaptive Stitching via Dynamics-Guided Trajectory Rollouts
• Learning Massively Multitask World Models for Continuous Control
• LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels
• Representation Learning Enables Scalable Multitask Deep Reinforcement Learning
• Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learning
• Transitive RL: Value Learning via Divide and Conquer
• Compositional Monte Carlo Tree Diffusion for Extendable Planning
• Optimistic Task Inference for Behavior Foundation Models
• Semantic World Models
• Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agents
• Expressive Value Learning for Scalable Offline Reinforcement Learning
• DEAS: DEtached value learning with Action Sequence for Scalable Offline RL
• Dual Goal Representations
• Test-Time Graph Search for Goal-Conditioned Reinforcement Learning
• TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learning
• SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modeling
• CREPE: Controlling Diffusion with Replica Exchange
• Unleashing Flow Policies with Distributional Critics
• Offline Goal-conditioned Reinforcement Learning with Quasimetric Representations
• RL²-VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models
• QQWorld: Quantile-Quantile Matching for World Model Regularization
• Good Rankers, Bad Objectives: Bilinear Contrastive Critics under Expressive Policy Search
• VH-Diffuser: Variable Horizon Diffusion Planner for Time-Aware Goal-Conditioned Trajectory Planning
• floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RL
• Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learning
• Tree-Guided Diffusion Planner
• Adaptive Scaling of Policy Constraints for Offline Reinforcement Learning
• One-Step Flow Q-Learning: Addressing the Diffusion Policy Bottleneck in Offline Reinforcement Learning
• GCHR : Goal-Conditioned Hindsight Regularization for Sample-Efficient Reinforcement Learning
• Benchmarking Massively Parallelized Multi-Task Reinforcement Learning for Robotics Tasks

Related

• Improving and Benchmarking Offline Reinforcement Learning Algorithms
• RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning
• SMORE: Score Models for Offline Goal-Conditioned Reinforcement Learning
• What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning