Hindsight Experience Replay

2017/07/05 by Marcin Andrychowicz, Andrychowicz, Marcin, Filip Wolski +17 · 264 citations
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #Artificial Intelligence (cs.AI) #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural and Evolutionary Computing (cs.NE) #Reinforcement Learning in Robotics #Robotics (cs.RO) #cs.AI #cs.LG #cs.NE #cs.RO

paper · pdf · doi:10.48550/arxiv.1707.01495

openalex publication_date 2017/07/05 · arxiv created 2018/02/23 · arxiv updated 2018/02/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Dealing with sparse rewards is one of the biggest challenges in Reinforcement Learning (RL). We present a novel technique called Hindsight Experience Replay which allows sample-efficient learning from rewards which are sparse and binary and therefore avoid the need for complicated reward engineering. It can be combined with an arbitrary off-policy RL algorithm and may be seen as a form of implicit curriculum. We demonstrate our approach on the task of manipulating objects with a robotic arm. In particular, we run experiments on three different tasks: pushing, sliding, and pick-and-place, in each case using only binary rewards indicating whether or not the task is completed. Our ablation studies show that Hindsight Experience Replay is a crucial ingredient which makes training possible in these challenging environments. We show that our policies trained on a physics simulation can be deployed on a physical robot and successfully complete the task.

Citations

• Learning and development in neural networks: the importance of starting small
• Prioritized Experience Replay
• Deep Exploration via Bootstrapped DQN
• Automated Curriculum Learning for Neural Networks
• Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World

Cited by

• Act2Goal: From World Model To General Goal-conditioned Policy
• Replay Failures as Successes: Sample-Efficient Reinforcement Learning for Instruction Following
• TEACH: Temporal Variance-Driven Curriculum for Reinforcement Learning
• Endpoint Replay: Compressing the Recency Buffer in Deep Reinforcement Learning
• MEMENTO: Memory-Guided Memetic Code-as-Policy Evolution
• Action-Sufficient Goal Representations
• Recontextualization Mitigates Specification Gaming without Modifying the Specification
• First-Order Representation Languages for Goal-Conditioned RL
• SD2AIL: Adversarial Imitation Learning from Synthetic Demonstrations via Diffusion Models
• Planning as Descent: Goal-Conditioned Latent Trajectory Synthesis in Learned Energy Landscapes
• Multivariate Uncertainty Quantification with Tomographic Quantile Forests
• Goal Reaching with Eikonal-Constrained Hierarchical Quasimetric Reinforcement Learning
• Enhancing Deep Deterministic Policy Gradients on Continuous Control Tasks with Decoupled Prioritized Experience Replay
• F2: Offline Reinforcement Learning for Hamiltonian Simulation via Free-Fermionic Subroutine Compilation
• Correspondence-Oriented Imitation Learning: Flexible Visuomotor Control with 3D Conditioning
• Driving Beyond Privilege: Distilling Dense-Reward Knowledge into Sparse-Reward Policies
• Zero-Shot Instruction Following in RL via Structured LTL Representations
• GR-RL: Going Dexterous and Precise for Long-Horizon Robotic Manipulation
• SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
• Hyper-GoalNet: Goal-Conditioned Manipulation Policy Learning with HyperNetworks
• Improving Language Agents through BREW: Bootstrapping expeRientially-learned Environmental knoWledge
• MagBotSim: Physics-Based Simulation and Reinforcement Learning Environments for Magnetic Robotics
• Beyond Fixed Tasks: Unsupervised Environment Design for Task-Level Pairs
• NFQ2.0: The CartPole Benchmark Revisited
• Expressive Temporal Specifications for Reward Monitoring
• Enabling Off-Policy Imitation Learning with Deep Actor Critic Stabilization
• Learning to Focus: Prioritizing Informative Histories with Structured Attention Mechanisms in Partially Observable Reinforcement Learning
• Physically-Grounded Goal Imagination: Physics-Informed Variational Autoencoder for Self-Supervised Reinforcement Learning
• Learning from Online Videos at Inference Time for Computer-Use Agents
• Forgetting is Everywhere
• Adaptable Hindsight Experience Replay for Search-Based Learning
• SLAP: Shortcut Learning for Abstract Planning
• Reinforcement Learning for Robotic Safe Control with Force Sensing
• CoRT: Counterfactual Replay for Token-Level Rubric-Guided Policy Optimization
• Estimating Q(s,s') with Deep Deterministic Dynamics Gradients
• ARCHER: Aggressive Rewards to Counter bias in Hindsight Experience Replay
• SymmGrid: Super-Scaling On-Robot Learning with Parallelized Symmetries and Egocentric-Exocentric Visual Perception
• Weakly-Supervised Reinforcement Learning for Controllable Behavior
• Automatic Goal Generation for Reinforcement Learning Agents
• D2 Actor Critic: Diffusion Actor Meets Distributional Critic
• Dense and Diverse Goal Coverage in Multi Goal Reinforcement Learning
• Multi-Robot Formation Control Using Reinforcement Learning
• Learning "Partner-Aware" Collaborators in Multi-Party Collaboration
• Confounding Robust Deep Reinforcement Learning: A Causal Approach
• A Unified Framework for Zero-Shot Reinforcement Learning
• DAIL: Beyond Task Ambiguity for Language-Conditioned Reinforcement Learning
• A Principle of Targeted Intervention for Multi-Agent Reinforcement Learning
• DDBot: Differentiable Physics-based Digging Robot for Unknown Granular Materials
• Consistent Zero-Shot Imitation with Contrastive Goal Inference
• RLAF: Reinforcement Learning from Automaton Feedback
• ARM-FM: Automated Reward Machines via Foundation Models for Compositional Reinforcement Learning
• Demystifying the Mechanisms Behind Emergent Exploration in Goal-conditioned RL
• A Forensic Analysis of Synthetic Data in RL: Diagnosing and Solving Algorithmic Failures in Model-Based Policy Optimization
• A Primer on SO(3) Action Representations in Deep Reinforcement Learning
• Reinforcement Learning with Action-Triggered Observations
• Sample-Efficient Online Learning in LM Agents via Hindsight Trajectory Rewriting
• Dejavu: Towards Experience Feedback Learning for Embodied Intelligence
• Towards Safe Maneuvering of Double-Ackermann-Steering Robots with a Soft Actor-Critic Framework
• Placeit! A Framework for Learning Robot Object Placement Skills
• Analytical Survey of Learning with Low-Resource Data: From Analysis to Investigation
• BaNEL: Exploration Posteriors for Generative Modeling Using Only Negative Rewards
• Agent Learning via Early Experience
• Hindsight Experience Replay with Kronecker Product Approximate Curvature
• Experiential Reinforcement Learning
• Test-Time Graph Search for Goal-Conditioned Reinforcement Learning
• Prompt reinforcing for long-term planning of large language models
• BuilderBench: The Building Blocks of Intelligent Agents
• Automaton Constrained Q-Learning
• Learning on the Job: Test-Time Curricula for Targeted Reinforcement Learning
• Learning to Act Through Contact: A Unified View of Multi-Task Robot Learning
• Aristotle: IMO-level Automated Theorem Proving
• Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewards
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• Training Agents using Upside-Down Reinforcement Learning
• Emergent Complexity via Multi-Agent Competition
• Efficient On-Policy Reinforcement Learning via Exploration of Sparse Parameter Space
• In-Context Compositional Q-Learning for Offline Reinforcement Learning
• SocialAI 0.1: Towards a Benchmark to Stimulate Research on Socio-Cognitive Abilities in Deep Reinforcement Learning Agents
• Offline Goal-conditioned Reinforcement Learning with Quasimetric Representations
• Outplaying elite table tennis players with an autonomous robot
• Sample Efficiency in Sparse Reinforcement Learning: Or Your Money Back
• Auditing Robot Learning for Safety and Compliance during Deployment
• Learning to Solve a Rubik's Cube with a Dexterous Hand
• Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few Demonstrations
• Sample Efficient Experience Replay in Non-stationary Environments
• Self-Paced Deep Reinforcement Learning
• LaGarNet: Goal-Conditioned Recurrent State-Space Models for Pick-and-Place Garment Flattening
• Generative Exploration and Exploitation
• Self-Supervised Goal-Reaching Results in Multi-Agent Cooperation and Exploration
• Robust Multi-Modal Policies for Industrial Assembly via Reinforcement Learning and Demonstrations: A Large-Scale Study
• Imagined Autocurricula
• Dexplore: Scalable Neural Control for Dexterous Manipulation from Reference-Scoped Exploration
• Universal Successor Features Approximators
• Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learning
• Reinforcement Learning with Anticipation: A Hierarchical Approach for Long-Horizon Tasks
• RoboBallet: Planning for Multi-Robot Reaching with Graph Neural Networks and Reinforcement Learning
• What Did You Think Would Happen? Explaining Agent Behaviour Through Intended Outcomes
• Reinforcement Learning with Time-dependent Goals for Robotic Musicians
• Autonomous Learning From Success and Failure: Goal-Conditioned Supervised Learning with Negative Feedback
• TAAC: Temporally Abstract Actor-Critic for Continuous Control
• Learning Latent Plans from Play
• HuBE: Cross-Embodiment Human-like Behavior Execution for Humanoid Robots
• Multi-task Reinforcement Learning with a Planning Quasi-Metric
• Universal Planning Networks
• Behavior From the Void: Unsupervised Active Pre-Training
• Goals and the Structure of Experience
• Unbiased Methods for Multi-Goal Reinforcement Learning
• Challenges of Context and Time in Reinforcement Learning: Introducing Space Fortress as a Benchmark
• Scaling All-Goals Updates in Reinforcement Learning Using Convolutional Neural Networks
• Developing a Simple Model for Sand-Tool Interaction and Autonomously Shaping Sand
• Visuomotor Grasping with World Models for Surgical Robots
• Scaling Up without Fading Out: Goal-Aware Sparse GNN for RL-based Generalized Planning
• Goal Discovery with Causal Capacity for Efficient Reinforcement Learning
• Towards Safe Imitation Learning via Potential Field-Guided Flow Matching
• ROLL: Visual Self-Supervised Reinforcement Learning with Object Reasoning
• ASkDAgger: Active Skill-level Data Aggregation for Interactive Imitation Learning
• A Deep Learning Approach to Grasping the Invisible
• Replay in Deep Learning: Current Approaches and Missing Biological Elements
• Causal Induction from Visual Observations for Goal Directed Tasks
• ACTRCE: Augmenting Experience via Teacher's Advice For Multi-Goal Reinforcement Learning
• RecoMind: A Reinforcement Learning Framework for Optimizing In-Session User Satisfaction in Recommendation Systems
• Dynamics-Aware Unsupervised Discovery of Skills
• Striving for Simplicity and Performance in Off-Policy DRL: Output Normalization and Non-Uniform Sampling
• Hierarchical Reinforcement Learning with Hindsight
• Adversarial Intrinsic Motivation for Reinforcement Learning
• Learning Generalizable Locomotion Skills with Hierarchical Reinforcement Learning
• Test-time Offline Reinforcement Learning on Goal-related Experience
• Task-Oriented Language Grounding for Language Input with Multiple Sub-Goals of Non-Linear Order
• Attention-Privileged Reinforcement Learning
• Effects of sparse rewards of different magnitudes in the speed of learning of model-based actor critic methods
• Efficient Robotic Task Generalization Using Deep Model Fusion Reinforcement Learning
• Sensor-Space Based Robust Kinematic Control of Redundant Soft Manipulator by Learning
• Universal Value Density Estimation for Imitation Learning and Goal-Conditioned Reinforcement Learning
• Divide-and-Conquer Reinforcement Learning
• A Survey of Reinforcement Learning Techniques: Strategies, Recent Development, and Future Directions
• Language-Conditioned Goal Generation: a New Approach to Language Grounding for RL
• PPAAS: PVT and Pareto Aware Analog Sizing via Goal-conditioned Reinforcement Learning
• Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities
• Model-free Reinforcement Learning for Model-based Control: Towards Safe, Interpretable and Sample-efficient Agents
• Self-Imitation Advantage Learning
• Reinforcement Learning for Industrial Control Network Cyber Security Orchestration
• V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
• Optimizing Drivers' Discount Order Acceptance Strategies: A Policy-Improved Deep Deterministic Policy Gradient Framework
• Hindsight Generative Adversarial Imitation Learning
• PlanGAN: Model-based Planning With Sparse Rewards and Multiple Goals
• DeepRacer: Educational Autonomous Racing Platform for Experimentation with Sim2Real Reinforcement Learning
• Evolutionary Stochastic Policy Distillation
• Mutual Information-based State-Control for Intrinsically Motivated Reinforcement Learning
• ELLA: Exploration through Learned Language Abstraction
• Variational Empowerment as Representation Learning for Goal-Based Reinforcement Learning
• A Survey of Reinforcement Learning for Software Engineering
• Learning The Minimum Action Distance
• Intelligent Control of Spacecraft Reaction Wheel Attitude Using Deep Reinforcement Learning
• Self-Improving Language Models for Evolutionary Program Synthesis: A Case Study on ARC-AGI
• 2048: Reinforcement Learning in a Delayed Reward Environment
• Hierarchical Reinforcement Learning with Targeted Causal Interventions
• Attribute-Based Robotic Grasping with One-Grasp Adaptation
• A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control
• Robotic self-representation improves manipulation skills and transfer learning
• Unsupervised Meta-Learning for Reinforcement Learning
• Zero-Incentive Dynamics: a look at reward sparsity through the lens of unrewarded subgoals
• Uncertainty Prioritized Experience Replay
• Differentiable Logic Machines
• Learning Obstacle Representations for Neural Motion Planning
• Maximum a Posteriori Policy Optimisation
• Forgetful Experience Replay in Hierarchical Reinforcement Learning from Demonstrations
• Strict Subgoal Execution: Reliable Long-Horizon Planning in Hierarchical Reinforcement Learning
• Hierarchical reinforcement learning for efficient exploration and transfer
• Flow-Based Single-Step Completion for Efficient and Expressive Policy Learning
• Efficient Skill Discovery via Regret-Aware Optimization
• Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction
• Learning Instruction-Following Policies through Open-Ended Instruction Relabeling with Large Language Models
• Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learning
• Using Logical Specifications of Objectives in Multi-Objective Reinforcement Learning
• Goal-oriented Trajectories for Efficient Exploration
• Offline Goal-Conditioned Reinforcement Learning with Projective Quasimetric Planning
• Reliability-Adjusted Prioritized Experience Replay
• Offline Meta-Reinforcement Learning with Online Self-Supervision
• Hierarchical Reinforcement Learning Framework towards Multi-agent Navigation
• A Survey of State Representation Learning for Deep Reinforcement Learning
• Modularization of End-to-End Learning: Case Study in Arcade Games
• BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning
• Energy-Based Transfer for Reinforcement Learning
• Reinforcement Learning with Augmented Data
• CAWR: Corruption-Averse Advantage-Weighted Regression for Robust Policy Optimization
• A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation
• HiLight: A Hierarchical Reinforcement Learning Framework with Global Adversarial Guidance for Large-Scale Traffic Signal Control
• TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization
• ClutterDexGrasp: A Sim-to-Real System for General Dexterous Grasping in Cluttered Scenes
• ViP: Video Platform for PyTorch
• Learning to Explore in Diverse Reward Settings via Temporal-Difference-Error Maximization
• DynaGuide: Steering Diffusion Polices with Active Dynamic Guidance
• Hierarchical Robot Navigation in Novel Environments using Rough 2-D Maps
• Goal-based Self-Adaptive Generative Adversarial Imitation Learning (Goal-SAGAIL) for Multi-goal Robotic Manipulation Tasks
• CIRO7.2: A Material Network with Circularity of -7.2 and Reinforcement-Learning-Controlled Robotic Disassembler
• Lifelong Robotic Reinforcement Learning by Retaining Experiences
• A Deeper Look at Experience Replay
• Reinforcement Learning Control of Robotic Knee with Human in the Loop by Flexible Policy Iteration
• SocialAI: Benchmarking Socio-Cognitive Abilities in Deep Reinforcement Learning Agents
• Learning Manipulation States and Actions for Efficient Non-prehensile Rearrangement Planning
• Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement
• Uncertainty-sensitive Learning and Planning with Ensembles
• Flexible and Efficient Long-Range Planning Through Curious Exploration
• Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation
• Reachability Weighted Offline Goal-conditioned Resampling
• SuperRL: Reinforcement Learning with Supervision to Boost Language Model Reasoning
• Hierarchical deep reinforcement learning controlled three-dimensional navigation of microrobots in blood vessels
• Placeto: Learning Generalizable Device Placement Algorithms for Distributed Machine Learning
• Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners
• Diffusion Guidance Is a Controllable Policy Improvement Operator
• When Autonomous Systems Meet Accuracy and Transferability through AI: A Survey
• Can Large Reasoning Models Self-Train?
• Hierarchical Reinforcement Learning with Uncertainty-Guided Diffusional Subgoals
• Evolving Rewards to Automate Reinforcement Learning
• Extremum Flow Matching for Offline Goal Conditioned Reinforcement Learning
• DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning
• Prompting Decision Transformers for Zero-Shot Reach-Avoid Policies
• HindSearch: Trajectory-Level Hindsight Critique for Search-Augmented Reinforcement Learning
• CiRL: Open-Source Environments for Reinforcement Learning in Circular Economy and Net Zero
• Improving On-policy Learning with Statistical Reward Accumulation
• Learning Adaptive Display Exposure for Real-Time Advertising
• Flattening Hierarchies with Policy Bootstrapping
• Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learning
• Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning
• Hindsight policy gradients
• GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms
• Attention-Based Reward Shaping for Sparse and Delayed Rewards
• Learning from Less: Guiding Deep Reinforcement Learning with Differentiable Symbolic Planning
• Electric Bus Charging Schedules Relying on Real Data-Driven Targets Based on Hierarchical Deep Reinforcement Learning
• Contrastive Active Inference
• Credit Assignment and Efficient Exploration based on Influence Scope in Multi-agent Reinforcement Learning
• UniSkill: Imitating Human Videos via Cross-Embodiment Skill Representations
• Asynchronous Episodic Deep Deterministic Policy Gradient: Towards Continuous Control in Computationally Complex Environments
• ADER: Adaptively Distilled Exemplar Replay Towards Continual Learning for Session-based Recommendation
• D3HRL: A Distributed Hierarchical Reinforcement Learning Approach Based on Causal Discovery and Spurious Correlation Detection
• A Goal-Oriented Reinforcement Learning-Based Path Planning Algorithm for Modular Self-Reconfigurable Satellites
• CAMOUFLAGE: Exploiting Misinformation Detection Systems Through LLM-driven Adversarial Claim Transformation
• Deep Reinforcement Learning: An Overview
• Motion Perception in Reinforcement Learning with Dynamic Objects
• Self-Paced Contextual Reinforcement Learning
• Where Did It Go Wrong? Process-Level Evaluation of Web Agents with Semantic State Tracking
• Learn Proportional Derivative Controllable Latent Space from Pixels
• MEMO: Memory-Augmented Model Context Optimization for Robust Multi-Turn Multi-Agent LLM Games
• Reward-Conditioned Reinforcement Learning
• Neuro-Symbolic Generation of Explanations for Robot Policies with Weighted Signal Temporal Logic
• unix-ctf: Procedural Environments for Unix-Competence Reinforcement Learning
• Replay-buffer engineering for noise-robust quantum circuit optimization
• Improving Zero-Shot Offline RL via Behavioral Task Sampling
• Hierarchical Reinforcement Learning in Multi-Goal Spatial Navigation with Autonomous Mobile Robots
• Efficient RL Training for LLMs with Experience Replay
• Locality-Sensitive Experience Replay for Online Recommendation
• Accelerating Reinforcement Learning with a Directional-Gaussian-Smoothing Evolution Strategy
• State2State: Environment-Derived Mid-Training for LLM Agents
• Accelerating Reinforcement Learning for Reaching Using Continuous Curriculum Learning
• Learning a Multi-Modal Policy via Imitating Demonstrations with Mixed Behaviors
• Planning with Diffusion Models for Target-Oriented Dialogue Systems
• Solving Sokoban using Hierarchical Reinforcement Learning with Landmarks
• Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning
• Diversity-based Trajectory and Goal Selection with Hindsight Experience Replay
• Don't Do What Doesn't Matter: Intrinsic Motivation with Action Usefulness
• t-soft update of target network for deep reinforcement learning
• Learning offline: memory replay in biological and artificial reinforcement learning
• DeepSlicing: Deep Reinforcement Learning Assisted Resource Allocation for Network Slicing
• Training a Conditioned Video Game Agent on a VLM Annotated Dataset
• Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models
• Next-Future: Sample-Efficient Policy Learning for Robotic-Arm Tasks
• Diffusion Models for Robotic Manipulation: A Survey
• Digital Twin Catalog: A Large-Scale Photorealistic 3D Object Digital Twin Dataset
• TW-CRL: Time-Weighted Contrastive Reward Learning for Efficient Inverse Reinforcement Learning

Related

• Learning and reusing primitive behaviours to improve Hindsight Experience Replay sample efficiency
• Improvements on Hindsight Learning
• Competitive Experience Replay
• Curiosity-Driven Multi-Criteria Hindsight Experience Replay
• Imaginary Hindsight Experience Replay: Curious Model-based Learning for Sparse Reward Tasks