2021/12/08 by Liran Szlak, Ohad Shamir, Szlak, Liran +1
Computer Science · Engineering · #Age of Information Optimization #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Ferroelectric and Negative Capacitance Devices #Machine Learning (cs.LG) #Stochastic Gradient Optimization Techniques #cs.AI #cs.LG
paper · pdf · doi:10.48550/arxiv.2112.04229
arxiv created 2021/12/08 · openalex publication_date 2021/12/08 · arxiv updated 2021/12/09 · openalex created_date 2022/11/14 · openalex updated_date 2026/07/28
Experience replay \citeplin1993reinforcement, mnih2015human is a widely used technique to achieve efficient use of data and improved performance in RL algorithms. In experience replay, past transitions are stored in a memory buffer and re-used during learning. Various suggestions for sampling schemes from the replay buffer have been suggested in previous works, attempting to optimally choose those experiences which will most contribute to the convergence to an optimal policy. Here, we give some conditions on the replay sampling scheme that will ensure convergence, focusing on the well-known Q-learning algorithm in the tabular setting. After establishing sufficient conditions for convergence, we turn to suggest a slightly different usage for experience replay - replaying memories in a biased manner as a means to change the properties of the resulting policy. We initiate a rigorous study of experience replay as a tool to control and modify the properties of the resulting policy. In particular, we show that using an appropriate biased sampling scheme can allow us to achieve a safe policy. We believe that using experience replay as a biasing mechanism that allows controlling the resulting policy in desirable ways is an idea with promising potential for many applications.