2020/10/20 by Benjamin Grimmer, Haihao Lu, Grimmer, Benjamin +5 · 1 citation
Computer Science · Neuroscience · Physics and Astronomy · #65K05 #65K10 #90C15 #90C26 #90C30 #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Model Reduction and Neural Networks #Neural dynamics and brain function #Optimization and Control (math.OC) #Stochastic Gradient Optimization Techniques
paper · pdf · doi:10.48550/arxiv.2010.10628
openalex publication_date 2020/10/20 · openalex created_date 2020/10/29 · openalex updated_date 2026/07/28
Unlike nonconvex optimization, where gradient descent is guaranteed to converge to a local optimizer, algorithms for nonconvex-nonconcave minimax optimization can have topologically different solution paths: sometimes converging to a solution, sometimes never converging and instead following a limit cycle, and sometimes diverging. In this paper, we study the limiting behaviors of three classic minimax algorithms: gradient descent ascent (GDA), alternating gradient descent ascent (AGDA), and the extragradient method (EGM). Numerically, we observe that all of these limiting behaviors can arise in Generative Adversarial Networks (GAN) training and are easily demonstrated for a range of GAN problems. To explain these different behaviors, we study the high-order resolution continuous-time dynamics that correspond to each algorithm, which results in the sufficient (and almost necessary) conditions for the local convergence by each method. Moreover, this ODE perspective allows us to characterize the phase transition between these different limiting behaviors caused by introducing regularization as Hopf Bifurcations.