2020/06/19 by Miguel Vázquez-Martin del Campo, Zhengxing Chen, Campo, Miguel +7
Computer Science · Engineering · #68T07 #Advanced Memory and Neural Computing #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #I.2.8 #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Reservoir Computing #Reinforcement Learning in Robotics
paper · pdf · doi:10.48550/arxiv.2006.11431
openalex publication_date 2020/06/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Soft Actor Critic (SAC) algorithms show remarkable performance in complex simulated environments. A key element of SAC networks is entropy regularization, which prevents the SAC actor from optimizing against fine grained features, oftentimes transient, of the state-action value function. This results in better sample efficiency during early training. We take this idea one step further by artificially bandlimiting the target critic spatial resolution through the addition of a convolutional filter. We derive the closed form solution in the linear case and show that bandlimiting reduces the interdependency between the low and high frequency components of the state-action value approximation, allowing the critic to learn faster. In experiments, the bandlimited SAC outperformed the classic twin-critic SAC in a number of Gym environments, and displayed more stability in returns. We derive novel insights about SAC by adding a stochastic noise disturbance, a technique that is increasingly being used to learn robust policies that transfer well to the real world counterparts.