2024/01/30 by Thomas Degris, Khurram Javed, Degris, Thomas +7
Engineering · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Image Processing Techniques and Applications #Indoor and Outdoor Localization Technologies #IoT-based Smart Home Systems #Machine Learning (cs.LG)
paper · pdf · doi:10.48550/arxiv.2401.17401
openalex publication_date 2024/01/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
In continual learning, a learner has to keep learning from the data over its whole life time. A key issue is to decide what knowledge to keep and what knowledge to let go. In a neural network, this can be implemented by using a step-size vector to scale how much gradient samples change network weights. Common algorithms, like RMSProp and Adam, use heuristics, specifically normalization, to adapt this step-size vector. In this paper, we show that those heuristics ignore the effect of their adaptation on the overall objective function, for example by moving the step-size vector away from better step-size vectors. On the other hand, stochastic meta-gradient descent algorithms, like IDBD (Sutton, 1992), explicitly optimize the step-size vector with respect to the overall objective function. On simple problems, we show that IDBD is able to consistently improve step-size vectors, where RMSProp and Adam do not. We explain the differences between the two approaches and their respective limitations. We conclude by suggesting that combining both approaches could be a promising future direction to improve the performance of neural networks in continual learning.