2019/01/30 by Clare Lyle, Pablo Samuel Castro, Lyle, Clare +3 · 3 citations
Computer Science · #Advanced Multi-Objective Optimization Algorithms #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics
paper · pdf · doi:10.48550/arxiv.1901.11084
openalex publication_date 2019/01/30 · openalex created_date 2022/07/30 · openalex updated_date 2026/07/28
Since their introduction a year ago, distributional approaches to\nreinforcement learning (distributional RL) have produced strong results\nrelative to the standard approach which models expected values (expected RL).\nHowever, aside from convergence guarantees, there have been few theoretical\nresults investigating the reasons behind the improvements distributional RL\nprovides. In this paper we begin the investigation into this fundamental\nquestion by analyzing the differences in the tabular, linear approximation, and\nnon-linear approximation settings. We prove that in many realizations of the\ntabular and linear approximation settings, distributional RL behaves exactly\nthe same as expected RL. In cases where the two methods behave differently,\ndistributional RL can in fact hurt performance when it does not induce\nidentical behaviour. We then continue with an empirical analysis comparing\ndistributional and expected RL methods in control settings with non-linear\napproximators to tease apart where the improvements from distributional RL\nmethods are coming from.\n