Planning with Goal-Conditioned Policies

2019/11/19 by Soroush Nasiriany, Vitchyr H. Pong, Nasiriany, Soroush +5 · 15 citations
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.1911.08453

openalex publication_date 2019/11/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Planning methods can solve temporally extended sequential decision making problems by composing simple behaviors. However, planning requires suitable abstractions for the states and transitions, which typically need to be designed by hand. In contrast, model-free reinforcement learning (RL) can acquire behaviors from low-level inputs directly, but often struggles with temporally extended tasks. Can we utilize reinforcement learning to automatically form the abstractions needed for planning, thus obtaining the best of both approaches? We show that goal-conditioned policies learned with RL can be incorporated into planning, so that a planner can focus on which states to reach, rather than how those states are reached. However, with complex state observations such as images, not all inputs represent valid states. We therefore also propose using a latent variable model to compactly represent the set of valid states for the planner, so that the policies provide an abstraction of actions, and the latent variable model provides an abstraction of states. We compare our method with planning-based and model-free methods and find that our method significantly outperforms prior work when evaluated on image-based robot navigation and manipulation tasks that require non-greedy, multi-staged behavior.

Cited by

• Action-Sufficient Goal Representations
• Correspondence-Oriented Imitation Learning: Flexible Visuomotor Control with 3D Conditioning
• Hyper-GoalNet: Goal-Conditioned Manipulation Policy Learning with HyperNetworks
• What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?
• Dense and Diverse Goal Coverage in Multi Goal Reinforcement Learning
• Is Temporal Difference Learning the Gold Standard for Stitching in RL?
• Demystifying the Mechanisms Behind Emergent Exploration in Goal-conditioned RL
• PoseDiff: A Unified Diffusion Model Bridging Robot Pose Estimation and Video-to-Action Control
• SAGE: Scene Graph-Aware Guidance and Execution for Long-Horizon Manipulation Tasks
• Solving Compositional Reinforcement Learning Problems via Task Reduction
• Goal Kernel Planning: Linearly-Solvable Non-Markovian Policies for Logical Tasks with Goal-Conditioned Options
• Autonomous Learning From Success and Failure: Goal-Conditioned Supervised Learning with Negative Feedback
• Multi-task Reinforcement Learning with a Planning Quasi-Metric
• C-Learning: Learning to Achieve Goals via Recursive Classification
• Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning

Related

• Simplified Temporal Consistency Reinforcement Learning
• Goal-Conditioned Reinforcement Learning with Imagined Subgoals
• Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Space
• An investigation of model-free planning
• Goal-Aware Prediction: Learning to Model What Matters