2014/10/11 by Biao Luo, Luo, Biao, Derong Liu +3
Computer Science · Engineering · #Adaptive Control of Nonlinear Systems #Adaptive Dynamic Programming Control #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (stat.ML) #Reinforcement Learning in Robotics #Systems and Control (eess.SY) #electronic engineering #information engineering
paper · pdf · doi:10.48550/arxiv.1410.2954
openalex publication_date 2014/10/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01
In this paper, two Q-learning (QL) methods are proposed and their convergence theories are established for addressing the model-free optimal control problem of general nonlinear continuous-time systems. By introducing the Q-function for continuous-time systems, policy iteration based QL (PIQL) and value iteration based QL (VIQL) algorithms are proposed for learning the optimal control policy from real system data rather than using mathematical system model. It is proved that both PIQL and VIQL methods generate a nonincreasing Q-function sequence, which converges to the optimal Q-function. For implementation of the QL algorithms, the method of weighted residuals is applied to derived the parameters update rule. The developed PIQL and VIQL algorithms are essentially off-policy reinforcement learning approachs, where the system data can be collected arbitrary and thus the exploration ability is increased. With the data collected from the real system, the QL methods learn the optimal control policy offline, and then the convergent control policy will be employed to real system. The effectiveness of the developed QL algorithms are verified through computer simulation.