Combining Model-Based and Model-Free Methods for Nonlinear Control: A Provably Convergent Policy Gradient Approach

2020/06/12 by Guannan Qu, Chenkai Yu, Qu, Guannan +5 · 2 citations
Computer Science · Engineering · Physics and Astronomy · #Adaptive Dynamic Programming Control #FOS: Computer and information sciences #FOS: Electrical engineering #FOS: Mathematics #Iterative Learning Control Systems #Machine Learning (cs.LG) #Model Reduction and Neural Networks #Optimization and Control (math.OC) #Systems and Control (eess.SY) #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2006.07476

openalex publication_date 2020/06/12 · openalex created_date 2020/06/19 · openalex updated_date 2026/08/04

Abstract

Model-free learning-based control methods have seen great success recently. However, such methods typically suffer from poor sample complexity and limited convergence guarantees. This is in sharp contrast to classical model-based control, which has a rich theory but typically requires strong modeling assumptions. In this paper, we combine the two approaches to achieve the best of both worlds. We consider a dynamical system with both linear and non-linear components and develop a novel approach to use the linear model to define a warm start for a model-free, policy gradient method. We show this hybrid approach outperforms the model-based controller while avoiding the convergence issues associated with model-free approaches via both numerical experiments and theoretical analyses, in which we derive sufficient conditions on the non-linear component such that our approach is guaranteed to converge to the (nearly) global optimal controller.

Citations

• End-to-End Training of Deep Visuomotor Policies
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Prediction and Control with Temporal Segment Models
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Learning-based Control of Unknown Linear Systems with Thompson Sampling
• Temporal Difference Models: Model-Free Deep RL for Model-Based Control
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
• Model-Based Reinforcement Learning via Meta-Policy Optimization
• Residual Policy Learning
• Stochastic Mirror Descent on Overparameterized Nonlinear Models: Convergence, Implicit Regularization, and Generalization
• Learning Linear-Quadratic Regulators Efficiently with only \√(T)\n Regret
• Learning-based Model Predictive Control for Safe Exploration
• Uncertainty-aware Model-based Policy Optimization
• LQR through the Lens of First Order Methods: Discrete-time Case
• Analyzing the Variance of Policy Gradient Estimators for the Linear-Quadratic Regulator
• Feedback Linearization for Unknown Systems via Reinforcement Learning
• Naive Exploration is Optimal for Online LQR

Cited by

• TAG-K: Tail-Averaged Greedy Kaczmarz for Computationally Efficient and Performant Online Inertial Parameter Estimation
• Reinforcement Twinning for Hybrid Control of Flapping-Wing Drones

Related

• Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator
• Accelerating Model-Free Policy Optimization Using Model-Based Gradient: A Composite Optimization Perspective
• Data Driven Control with Learned Dynamics: Model-Based versus Model-Free Approach
• Feedback Linearization for Unknown Systems via Reinforcement Learning
• A Model-Free Optimal Control Method With Fixed Terminal States and Delay