2019/12/20 by Jingbin Liu, Shuai Liu, Liu, Jingbin +3 · 1 citation
Computer Science · Decision Sciences · Mathematics · #Advanced Bandit Algorithms Research #Artificial Intelligence (cs.AI) #Artificial intelligence #Backup #Computer science #Computer security #Data Stream Mining Techniques #Database #Distributed computing #Equivalence (formal languages) #Exploit #FOS: Computer and information sciences #Machine Learning (cs.LG) #Mathematics #Regularization (linguistics) #Reinforcement Learning in Robotics #Reinforcement learning #Theoretical computer science #cs.AI #cs.LG
paper · pdf · doi:10.48550/arxiv.1912.10891
openalex publication_date 2019/12/20 · arxiv created 2020/12/14 · arxiv updated 2020/12/15 · openalex created_date 2024/04/11 · openalex updated_date 2026/07/28
Deep Q Network (DQN) is a very successful algorithm, yet the inherent problem of reinforcement learning, i.e. the exploit-explore balance, remains. In this work, we introduce entropy regularization into DQN and propose SQN. We find that the backup equation of soft Q learning can enjoy the corrective feedback if we view the soft backup as policy improvement in the form of Q, instead of policy evaluation. We show that Soft Q Learning with Corrective Feedback (SQL-CF) underlies the on-plicy nature of SQL and the equivalence of SQL and Soft Policy Gradient (SPG). With these insights, we propose an on-policy version of deep Q learning algorithm, i.e. Q On-Policy (QOP). We experiment with QOP on a self-play environment called Google Research Football (GRF). The QOP algorithm exhibits great stability and efficiency in training GRF agents.