2017/11/21 by Daniel Levy, Stefano Ermon, Levy, Daniel +1
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics
paper · pdf · doi:10.48550/arxiv.1711.08068
openalex publication_date 2017/11/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Policy optimization methods have shown great promise in solving complex\nreinforcement and imitation learning tasks. While model-free methods are\nbroadly applicable, they often require many samples to optimize complex\npolicies. Model-based methods greatly improve sample-efficiency but at the cost\nof poor generalization, requiring a carefully handcrafted model of the system\ndynamics for each task. Recently, hybrid methods have been successful in\ntrading off applicability for improved sample-complexity. However, these have\nbeen limited to continuous action spaces. In this work, we present a new hybrid\nmethod based on an approximation of the dynamics as an expectation over the\nnext state under the current policy. This relaxation allows us to derive a\nnovel hybrid policy gradient estimator, combining score function and pathwise\nderivative estimators, that is applicable to discrete action spaces. We show\nsignificant gains in sample complexity, ranging between 1.7 and 25\×,\nwhen learning parameterized policies on Cart Pole, Acrobot, Mountain Car and\nHand Mass. Our method is applicable to both discrete and continuous action\nspaces, when competing pathwise methods are limited to the latter.\n