2020/06/12 by Harshat Kumar, Dionysios S. Kalogerias, Kumar, Harshat +5 · 5 citations
Computer Science · Mathematics · #Action (physics) #Adversarial Robustness in Machine Learning #Applied mathematics #Artificial intelligence #Computer science #Economics #FOS: Computer and information sciences #FOS: Mathematics #Function (biology) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine learning #Markov decision process #Markov process #Mathematical economics #Mathematical optimization #Mathematics #Optimization and Control (math.OC) #Order (exchange) #Physics #Randomness #Reinforcement Learning in Robotics #Reinforcement learning #Sample (material) #Stability (learning theory) #Statistical physics #Statistics #Stochastic Gradient Optimization Techniques #cs.LG #math.OC #stat.ML
paper · pdf · doi:10.48550/arxiv.2006.07314
published in arXiv (Cornell University) (Cornell University) · 18 pages, 5 figures. Fixed some minor oversights in the theoretical development present in the previous version of the manuscript and significantly revised and expanded the simulations sections, both in the main body and supplementary material
openalex publication_date 2020/06/12 · arxiv created 2020/07/11 · arxiv updated 2020/07/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
Deterministic Policy Gradient (DPG) removes a level of randomness from standard randomized-action Policy Gradient (PG), and demonstrates substantial empirical success for tackling complex dynamic problems involving Markov decision processes. At the same time, though, DPG loses its ability to learn in a model-free (i.e., actor-only) fashion, frequently necessitating the use of critics in order to obtain consistent estimates of the associated policy-reward gradient. In this work, we introduce Zeroth-order Deterministic Policy Gradient (ZDPG), which approximates policy-reward gradients via two-point stochastic evaluations of the Q-function, constructed by properly designed low-dimensional action-space perturbations. Exploiting the idea of random horizon rollouts for obtaining unbiased estimates of the Q-function, ZDPG lifts the dependence on critics and restores true model-free policy learning, while enjoying built-in and provable algorithmic stability. Additionally, we present new finite sample complexity bounds for ZDPG, which improve upon existing results by up to two orders of magnitude. Our findings are supported by several numerical experiments, which showcase the effectiveness of ZDPG in a practical setting, and its advantages over both PG and Baseline PG.