2019/11/05 by Jonas Degrave, Abbas Abdolmaleki, Degrave, Jonas +7 · 2 citations
Computer Science · Engineering · #Advanced Thermodynamic Systems and Engines #Control Systems and Identification #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural and Evolutionary Computing (cs.NE) #cs.LG #cs.NE
paper · pdf · doi:10.48550/arxiv.1911.01831
Deep RL Workshop/NeurIPS
arxiv created 2019/11/05 · openalex publication_date 2019/11/05 · arxiv updated 2019/11/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
We present an algorithm for learning an approximate action-value soft Q-function in the relative entropy regularised reinforcement learning setting, for which an optimal improved policy can be recovered in closed form. We use recent advances in normalising flows for parametrising the policy together with a learned value-function; and show how this combination can be used to implicitly represent Q-values of an arbitrary policy in continuous action space. Using simple temporal difference learning on the Q-values then leads to a unified objective for policy and value learning. We show how this approach considerably simplifies standard Actor-Critic off-policy algorithms, removing the need for a policy optimisation step. We perform experiments on a range of established reinforcement learning benchmarks, demonstrating that our approach allows for complex, multimodal policy distributions in continuous action spaces, while keeping the process of sampling from the policy both fast and exact.