Training Generative Adversarial Networks with Adaptive Composite Gradient

2021/11/10 by Huiqing Qi, Qi Huiqing, Qi, Huiqing +7
Computer Science · Mathematics · #Advanced Vision and Imaging #Artificial Intelligence (cs.AI) #Computer Science and Game Theory (cs.GT) #Dynamical Systems (math.DS) #FOS: Computer and information sciences #FOS: Mathematics #Generative Adversarial Networks and Image Synthesis #Human Pose and Action Recognition #Machine Learning (cs.LG) #Optimization and Control (math.OC) #cs.AI #cs.GT #cs.LG #math.DS #math.OC

paper · pdf · doi:10.48550/arxiv.2111.05508

arxiv created 2021/11/10 · openalex publication_date 2021/11/10 · arxiv updated 2021/11/11 · openalex created_date 2021/11/22 · openalex updated_date 2026/07/28

Abstract

The wide applications of Generative adversarial networks benefit from the successful training methods, guaranteeing that an object function converges to the local minima. Nevertheless, designing an efficient and competitive training method is still a challenging task due to the cyclic behaviors of some gradient-based ways and the expensive computational cost of these methods based on the Hessian matrix. This paper proposed the adaptive Composite Gradients (ACG) method, linearly convergent in bilinear games under suitable settings. Theory and toy-function experiments suggest that our approach can alleviate the cyclic behaviors and converge faster than recently proposed algorithms. Significantly, the ACG method is not only used to find stable fixed points in bilinear games as well as in general games. The ACG method is a novel semi-gradient-free algorithm since it does not need to calculate the gradient of each step, reducing the computational cost of gradient and Hessian by utilizing the predictive information in future iterations. We conducted two mixture of Gaussians experiments by integrating ACG to existing algorithms with Linear GANs. Results show ACG is competitive with the previous algorithms. Realistic experiments on four prevalent data sets (MNIST, Fashion-MNIST, CIFAR-10, and CelebA) with DCGANs show that our ACG method outperforms several baselines, which illustrates the superiority and efficacy of our method.

Citations

• Generative Adversarial Networks
• Gradient-based learning applied to document recognition
• Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimization
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• DualGAN: Unsupervised Dual Learning for Image-to-Image Translation
• Voice Conversion from Unaligned Corpora using Variational Autoencoding Wasserstein Generative Adversarial Networks
• Objective-Reinforced Generative Adversarial Networks (ORGAN) for Sequence Generation Models
• Stabilizing GAN Training with Multiple Random Projections
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• First-order Methods Almost Always Avoid Saddle Points
• A Variational Inequality Perspective on Generative Adversarial Networks
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Training Generative Adversarial Networks by Solving Ordinary Differential Equations
• Learning Multiple Layers of Features from Tiny Images

Related

• Composite Functional Gradient Learning of Generative Adversarial Models
• The Numerics of GANs
• An Online Learning Approach to Generative Adversarial Networks
• A Survey on Generative Adversarial Networks: Variants, Applications, and\n Training
• Training Generative Adversarial Networks via Primal-Dual Subgradient Methods: A Lagrangian Perspective on GAN