Convergence and Sample Complexity of SGD in GANs

2020/12/01 by Kontonis, Vasilis, Liu, Sihan, Tzamos, Christos
#FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Statistics Theory (math.ST)

paper · doi:10.48550/arxiv.2012.00732

Abstract

We provide theoretical convergence guarantees on training Generative Adversarial Networks (GANs) via SGD. We consider learning a target distribution modeled by a 1-layer Generator network with a non-linear activation function ϕ(⋅) parametrized by a d × d weight matrix \mathbf W_*, i.e., f_*(\mathbf x) = ϕ(\mathbf W_* \mathbf x). Our main result is that by training the Generator together with a Discriminator according to the Stochastic Gradient Descent-Ascent iteration proposed by Goodfellow et al. yields a Generator distribution that approaches the target distribution of f_*. Specifically, we can learn the target distribution within total-variation distance ε using O(d²/ε²) samples which is (near-)information theoretically optimal. Our results apply to a broad class of non-linear activation functions ϕ, including ReLUs and is enabled by a connection with truncated statistics and an appropriate design of the Discriminator network. Our approach relies on a bilevel optimization framework to show that vanilla SGDA works.

Related

• SGD Learns One-Layer Networks in WGANs
• Forward Super-Resolution: How Can GANs Learn Hierarchical Generative Models for Real-World Distributions
• SAN: Inducing Metrizability of GAN with Discriminative Normalized Linear Layer
• Deconstructing Generative Adversarial Networks
• Improving Generalization and Stability of Generative Adversarial Networks