Policy Optimization for Unknown Systems using Differentiable Model Predictive Control

2025/11/14 by Riccardo Zuliani, Efe C. Balta, Zuliani, Riccardo +3
Computer Science · Engineering · #Adaptive Dynamic Programming Control #Advanced Control Systems Optimization #FOS: Electrical engineering #FOS: Mathematics #Optimization and Control (math.OC) #Stability and Control of Uncertain Systems #Systems and Control (eess.SY) #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2511.11308

openalex publication_date 2025/11/14 · openalex created_date 2025/11/18 · openalex updated_date 2026/07/28

Abstract

Model-based policy optimization often struggles with inaccurate system dynamics models, leading to suboptimal closed-loop performance. This challenge is especially evident in Model Predictive Control (MPC) policies, which rely on the model for real-time trajectory planning and optimization. We introduce a novel policy optimization framework for MPC-based policies combining differentiable optimization with zeroth-order optimization. Our method combines model-based and model-free gradient estimation approaches, achieving faster transient performance compared to fully data-driven approaches while maintaining convergence guarantees, even under model uncertainty. We demonstrate the effectiveness of the proposed approach on a nonlinear control task involving a 12-dimensional quadcopter model.

Citations

• Differentiable-by-design Nonlinear Optimization for Model Predictive Control
• Gray-Box Nonlinear Feedback Optimization
• BP-MPC: Optimizing the Closed-Loop Performance of MPC using BackPropagation
• Goldstein Stationarity in Lipschitz Constrained Optimization
• Gradient-Free Methods for Deterministic and Stochastic Nonsmooth Nonconvex Optimization
• Deep Learning Alternative to Explicit Model Predictive Control for Unknown Nonlinear Systems
• Learning Convex Optimization Control Policies
• Conservative set valued fields, automatic differentiation, stochastic gradient method and deep learning
• Differentiable MPC for End-to-end Planning and Control
• Stochastic subgradient method converges on tame functions
• An Optimal Algorithm for Bandit and Zero-Order Convex Optimization with Two-Point Feedback
• Randomized Smoothing for Stochastic Optimization
• Online convex optimization in the bandit setting: gradient descent without a gradient

Related

• Policy Search for Model Predictive Control with Application for Agile Drone Flight
• Rollout Then Optimize: A One-Step Newton Refinement of Learned Policies for Nonlinear Model Predictive Control
• Model-Agnostic Meta Learning for Differentiable MPC
• Differentiable-by-design Nonlinear Optimization for Model Predictive Control
• Differentiable Robust Model Predictive Control