Temporal Difference Learning for Model Predictive Control

2022/03/09 by Nicklas Hansen, Hansen, Nicklas, Xiaolong Wang +3 · 124 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · Engineering · #Advanced Control Systems Optimization #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Algorithms #Metabolomics and Mass Spectrometry Studies #Robotics (cs.RO) #cs.LG #cs.RO

paper · pdf · doi:10.48550/arxiv.2203.04955

Code and videos: https://nicklashansen.github.io/td-mpc

openalex publication_date 2022/03/09 · arxiv created 2022/07/19 · arxiv updated 2022/07/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Data-driven model predictive control has two key advantages over model-free methods: a potential for improved sample efficiency through model learning, and better performance as computational budget for planning increases. However, it is both costly to plan over long horizons and challenging to obtain an accurate model of the environment. In this work, we combine the strengths of model-free and model-based methods. We use a learned task-oriented latent dynamics model for local trajectory optimization over a short horizon, and use a learned terminal value function to estimate long-term return, both of which are learned jointly by temporal difference learning. Our method, TD-MPC, achieves superior sample efficiency and asymptotic performance over prior work on both state and image-based continuous control tasks from DMControl and Meta-World. Code and video results are available at https://nicklashansen.github.io/td-mpc.

Cited by

• Sampling-Based Optimization with Parallelized Physics Simulator for Bimanual Manipulation
• Embodied Robot Manipulation in the Era of Foundation Models: Planning and Learning Perspectives
• Action from Adjacent Set in Physical Space Outperforms the Best Prediction in World Models
• Biased Dreams: Limitations to Epistemic Uncertainty Quantification in Latent Dynamics Models
• Spectral Representation-based Reinforcement Learning
• Double Horizon Model-Based Policy Optimization
• CORL: Reinforcement Learning of MILP Policies Solved via Branch and Bound
• Prismatic World Model: Learning Compositional Dynamics for Planning in Hybrid Systems
• GR-RL: Going Dexterous and Precise for Long-Horizon Robotic Manipulation
• Learning Massively Multitask World Models for Continuous Control
• Learning Human-Like RL Agents Through Trajectory Optimization With Action Quantization
• CoRL-MPPI: Enhancing MPPI With Learnable Behaviours For Efficient And Provably-Safe Multi-Robot Collision Avoidance
• LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels
• Representation Learning Enables Scalable Multitask Deep Reinforcement Learning
• Model Predictive Control via Probabilistic Inference: A Tutorial
• Next-Latent Prediction Transformers Learn Compact World Models
• WorldPlanner: Monte Carlo Tree Search and MPC with Action-Conditioned Visual World Models
• Learning Interactive World Model for Object-Centric Reinforcement Learning
• Bootstrap Off-policy with World Model
• A Step Toward World Models: A Survey on Robotic Manipulation
• Off-policy Reinforcement Learning with Model-based Exploration Augmentation
• D2 Actor Critic: Diffusion Actor Meets Distributional Critic
• Visual Generation Unlocks Human-Like Reasoning through Multimodal World Models
• Mixed-Density Diffuser: Efficient Planning with Non-Uniform Temporal Resolution
• World Models Should Prioritize the Unification of Physical and Social Dynamics
• Social World Model-Augmented Mechanism Design Policy Learning
• Semantic World Models
• Bi-Level Reinforcement Learning Pathway for Sim-to-Real Optimality
• Learning to Design Soft Hands using Reward Models
• A Recipe for Efficient Sim-to-Real Transfer in Manipulation with Online Imitation-Pretrained World Models
• Ego-Vision World Model for Humanoid Contact Planning
• MatterDoor: Sampling Zero-shot Spatio-semantic Priors using Generative Models
• Ctrl-World: A Controllable Generative World Model for Robot Manipulation
• Model-Based Lookahead Reinforcement Learning for in-hand manipulation
• Expressive Value Learning for Scalable Offline Reinforcement Learning
• AVO: Amortized Value Optimization for Contact Mode Switching in Multi-Finger Manipulation
• Generative World Modelling for Humanoids: 1X World Model Challenge Technical Report
• Test-Time Graph Search for Goal-Conditioned Reinforcement Learning
• From Pixels to Factors: Learning Independently Controllable State Variables for Reinforcement Learning
• A KL-regularization Framework for Learning to Plan with Adaptive Priors
• Model-Based Reinforcement Learning under Random Observation Delays
• World Action Planner: Generalizable Decision-Making with Action-Conditioned World Models
• ODEWorld: A Continuous Predictive Architecture via Physical-Time Flow
• Temporal Straightening for Latent Planning
• High-Precision and High-Efficiency Trajectory Tracking for Excavators Based on Closed-Loop Dynamics
• Empowering Multi-Robot Cooperation via Sequential World Models
• Why and How Auxiliary Tasks Improve JEPA Representations
• One Model for All Tasks: Leveraging Efficient World Models in Multi-Task Planning
• floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RL
• RT-HCP: Dealing with Inference Delays and Sample Efficiency to Learn Directly on Robotic Platforms
• Learning to Walk with Less: a Dyna-Style Approach to Quadrupedal Locomotion
• Grasp-MPC: Closed-Loop Visual Grasping via Value-Guided Model Predictive Control
• First Order Model-Based RL through Decoupled Backpropagation
• Neural Robot Dynamics
• Action-Constrained Imitation Learning
• Self-Predictive Representations for Combinatorial Generalization in Behavioral Cloning
• Improving Pre-Trained Vision-Language-Action Policies with Model-Based Search
• Learning State-Space Models of Dynamic Systems from Arbitrary Data using Joint Embedding Predictive Architectures
• Latent Policy Barrier: Learning Robust Visuomotor Policies by Staying In-Distribution
• ReSim: Reliable World Simulation for Autonomous Driving
• DiWA: Diffusion Policy Adaptation with World Models
• Is Exploration or Optimization the Problem for Deep Reinforcement Learning?
• Model Predictive Adversarial Imitation Learning for Planning from Observation
• Revisiting Bisimulation Metric for Robust Representations in Reinforcement Learning
• LeRobot: An Open-Source Library for End-to-End Robot Learning
• V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
• Time-Aware World Model for Adaptive Prediction and Control
• MOBODY: Model Based Off-Dynamics Offline Reinforcement Learning
• Explore Beyond the Boundary Using Entropic Information
• Q-Guided Stein Variational Model Predictive Control via RL-informed Policy Prior
• Sample-Efficient Reinforcement Learning Controller for Deep Brain Stimulation in Parkinson's Disease
• FBFM: A Training-Free Asynchronous Feedback Mechanism for Flow-Matching in World-Action Models Execution
• TD-MPC-Opt: Distilling Model-Based Multi-Task Reinforcement Learning Agents
• Curious Causality-Seeking Agents Learn Meta Causal World
• M3PO: Massively Multi-Task Model-Based Policy Optimization
• Real-Time Execution of Action Chunking Flow Policies
• Investigating Lagrangian Neural Networks for Infinite Horizon Planning in Quadrupedal Locomotion
• Distributions as Actions: A Unified Framework for Diverse Action Spaces
• AMPLIFY: Actionless Motion Priors for Robot Learning from Videos
• DynaGuide: Steering Diffusion Polices with Active Dynamic Guidance
• DoublyAware: Dual Planning and Policy Awareness for Temporal Difference Learning in Humanoid Locomotion
• CLAM: Continuous Latent Action Models for Robot Learning from Unlabeled Demonstrations
• BiTrajDiff: Bidirectional Trajectory Generation with Diffusion Models for Offline Reinforcement Learning
• A Smooth Sea Never Made a Skilled SAILOR: Robust Imitation via Learning to Search
• Simple, Good, Fast: Self-Supervised World Models Free of Baggage
• WoMAP: World Models For Embodied Open-Vocabulary Object Localization
• SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics
• Calibrated Value-Aware Model Learning with Probabilistic Environment Models
• Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspective
• OSVI-WM: One-Shot Visual Imitation for Unseen Tasks using World-Model-Guided Trajectory Generation
• DreamPolicy: A Unified World-model Policy for Scalable Humanoid Locomotion
• Is Single-View Mesh Reconstruction Ready for Robotics?
• Maximum Total Correlation Reinforcement Learning
• Enter the Void - Planning to Seek Entropy When Reward is Scarce
• Hadamax Encoding: Elevating Performance in Model-Free Atari
• FLARE: Robot Learning with Implicit World Modeling
• World Models as Reference Trajectories for Rapid Motor Adaptation
• TD-GRPC: Temporal Difference Learning with Group Relative Policy Constraint for Humanoid Locomotion
• Certifying Stability of Reinforcement Learning Policies using Generalized Lyapunov Functions
• Accelerating Visual-Policy Learning through Parallel Differentiable Simulation
• FlowDreamer: A RGB-D World Model with Flow-based Motion Representations for Robot Manipulation
• Cache-Efficient Posterior Sampling for Reinforcement Learning with LLM-Derived Priors Across Discrete and Continuous Domains
• Pretraining a Shared Q-Network for Data-Efficient Offline Reinforcement Learning
• Coupled Distributional Random Expert Distillation for World Model Online Imitation Learning
• AdaJEPA: An Adaptive Latent World Model
• A Lightweight Library for Energy-Based Joint-Embedding Predictive Architectures
• Fast LeWorldModel
• Beyond Language Modeling: An Exploration of Multimodal Pretraining
• Next Embedding Prediction Makes World Models Stronger
• Multi-Goal Dexterous Hand Manipulation using Probabilistic Model-based Reinforcement Learning
• On Training in Imagination
• Hallucination in World Models is Predictable and Preventable
• WEAVER, Better, Faster, Longer: An Effective World Model for Robotic Manipulation
• World-Value-Action Model: Implicit Planning for Vision-Language-Action Systems
• RoboVerse: Towards a Unified Platform, Dataset and Benchmark for Scalable and Generalizable Robot Learning
• PIN-WM: Learning Physics-INformed World Models for Non-Prehensile Manipulation
• Solving New Tasks by Adapting Internet Video Knowledge
• Enhanced Pruning Strategy for Multi-Component Neural Architectures Using Component-Aware Graph Analysis
• MInCo: Mitigating Information Conflicts in Distracted Visual Model-based Reinforcement Learning
• Dream-MPC: Gradient-Based Model Predictive Control with Latent Imagination
• Compositional Flows for 3D Molecule and Synthesis Pathway Co-design
• Fast Adaptation with Behavioral Foundation Models
• Neural Motion Simulator: Pushing the Limit of World Models in Reinforcement Learning
• Trust-Region Twisted Policy Improvement

Related

• Temporal Difference Models: Model-Free Deep RL for Model-Based Control
• Model Predictive Control with Self-supervised Representation Learning
• Tractable Stochastic Hybrid Model Predictive Control using Gaussian Processes for Repetitive Tasks in Unseen Environments
• Reinforcement Learning of the Prediction Horizon in Model Predictive Control
• Bootstrapped Model Predictive Control