Anderson Acceleration for Reinforcement Learning

2018/09/25 by Geist, Matthieu, Bruno Scherrer, Scherrer, Bruno · 8 citations
Computer Science · #Adaptive Dynamic Programming Control #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.1809.09501

openalex publication_date 2018/09/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Anderson acceleration is an old and simple method for accelerating the computation of a fixed point. However, as far as we know and quite surprisingly, it has never been applied to dynamic programming or reinforcement learning. In this paper, we explain briefly what Anderson acceleration is and how it can be applied to value iteration, this being supported by preliminary experiments showing a significant speed up of convergence, that we critically discuss. We also discuss how this idea could be applied more generally to (deep) reinforcement learning.

Cited by

• Spectral Analysis of Heavy-Ball Q-value Iteration
• Control and Reinforcement Learning through the Lens of Optimization: An Algorithmic Perspective
• Accelerated implicitization: Robust fixed-point iterations arising from an explicit scheme
• A Generalized Alternating Anderson Acceleration Method
• On Anderson acceleration for partially observable Markov decision processes
• Nonmonotone Globalization for Anderson Acceleration via Adaptive Regularization
• Damped Anderson Mixing for Deep Reinforcement Learning: Acceleration, Convergence, and Stabilization
• Rank-One Modified Value Iteration

Related

• Damped Anderson Mixing for Deep Reinforcement Learning: Acceleration, Convergence, and Stabilization
• Regularized Anderson Acceleration for Off-Policy Deep Reinforcement Learning
• Anderson acceleration with adaptive relaxation for convergent fixed-point iterations
• Acceleration methods for fixed point iterations
• Stable Anderson Acceleration for Deep Learning