WGAN with an Infinitely Wide Generator Has No Spurious Stationary Points

2021/02/15 by Albert No, Taeho Yoon, No, Albert +6
Computer Science · Mathematics · Physics and Astronomy · #FOS: Computer and information sciences #FOS: Mathematics #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Model Reduction and Neural Networks #Neural Networks and Applications #Optimization and Control (math.OC) #cs.LG #math.OC

paper · pdf · doi:10.48550/arxiv.2102.07541

Published at ICML 2021

openalex publication_date 2021/02/15 · arxiv created 2021/06/09 · arxiv updated 2021/06/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Generative adversarial networks (GAN) are a widely used class of deep generative models, but their minimax training dynamics are not understood very well. In this work, we show that GANs with a 2-layer infinite-width generator and a 2-layer finite-width discriminator trained with stochastic gradient ascent-descent have no spurious stationary points. We then show that when the width of the generator is finite but wide, there are no spurious stationary points within a ball whose radius becomes arbitrarily large (to cover the entire parameter space) as the width goes to infinity.

Citations

• Approximation by superpositions of a sigmoidal function
• NIPS 2016 Tutorial: Generative Adversarial Networks
• How to Escape Saddle Points Efficiently
• Understanding the Loss Surface of Neural Networks for Binary Classification
• On the Convergence and Robustness of Training GANs with Regularized Optimal Transport
• Adding One Neuron Can Eliminate All Bad Local Minima
• On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport
• Learning Overparameterized Neural Networks via Stochastic Gradient\n Descent on Structured Data
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition
• When Does Non-Orthogonal Tensor Decomposition Have No Spurious Local Minima?

Related

• Instability and Local Minima in GAN Training with Kernel Discriminators
• Deconstructing Generative Adversarial Networks
• GAN-QP: A Novel GAN Framework without Gradient Vanishing and Lipschitz Constraint
• Convergence Problems with Generative Adversarial Networks (GANs)
• Improving Generalization and Stability of Generative Adversarial Networks