2023/06/18 by Mudit Gaur, Amrit Singh Bedi, Gaur, Mudit +5 · 2 citations
Computer Science · #Adversarial Robustness in Machine Learning #F.2.1 #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural Networks and Reservoir Computing #Reinforcement Learning in Robotics
paper · pdf · doi:10.48550/arxiv.2306.10486
openalex publication_date 2023/06/18 · openalex created_date 2023/06/22 · openalex updated_date 2026/07/28
Actor-critic algorithms have shown remarkable success in solving state-of-the-art decision-making problems. However, despite their empirical effectiveness, their theoretical underpinnings remain relatively unexplored, especially with neural network parametrization. In this paper, we delve into the study of a natural actor-critic algorithm that utilizes neural networks to represent the critic. Our aim is to establish sample complexity guarantees for this algorithm, achieving a deeper understanding of its performance characteristics. To achieve that, we propose a Natural Actor-Critic algorithm with 2-Layer critic parametrization (NAC2L). Our approach involves estimating the Q-function in each iteration through a convex optimization problem. We establish that our proposed approach attains a sample complexity of O(\frac1ε4(1-γ)4). In contrast, the existing sample complexity results in the literature only hold for a tabular or linear MDP. Our result, on the other hand, holds for countable state spaces and does not require a linear or low-rank structure on the MDP.