2021/01/27 by Philip Ball, Stephen Roberts, Ball, Philip J. +1
Decision Sciences · Engineering · Physics and Astronomy · #Advanced Bandit Algorithms Research #Advanced Control Systems Optimization #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Model Reduction and Neural Networks #Optimization and Control (math.OC)
paper · pdf · doi:10.48550/arxiv.2101.11331
openalex publication_date 2021/01/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Two popular approaches to model-free continuous control tasks are SAC and TD3. At first glance these approaches seem rather different; SAC aims to solve the entropy-augmented MDP by minimising the KL-divergence between a stochastic proposal policy and a hypotheical energy-basd soft Q-function policy, whereas TD3 is derived from DPG, which uses a deterministic policy to perform policy gradient ascent along the value function. In reality, both approaches are remarkably similar, and belong to a family of approaches we call `Off-Policy Continuous Generalized Policy Iteration'. This illuminates their similar performance in most continuous control benchmarks, and indeed when hyperparameters are matched, their performance can be statistically indistinguishable. To further remove any difference due to implementation, we provide OffCon3 (Off-Policy Continuous Control: Consolidated), a code base featuring state-of-the-art versions of both algorithms.