A Decentralized Parallel Algorithm for Training Generative Adversarial Nets

2019/10/28 by Mingrui Liu, Wei Zhang, Liu, Mingrui +11 · 34 citations
Computer Science · Mathematics · #Adversarial system #Algorithm #Artificial intelligence #Computer science #Distributed computing #FOS: Computer and information sciences #FOS: Mathematics #Generative Adversarial Networks and Image Synthesis #Generative grammar #Geography #Human Pose and Action Recognition #Machine Learning (cs.LG) #Optimization and Control (math.OC) #Theoretical computer science #Training (meteorology) #Video Surveillance and Tracking Methods #cs.LG #math.OC

paper · pdf · doi:10.48550/arxiv.1910.12999

published in arXiv (Cornell University) (Cornell University) · Accepted by NeurIPS 2020

openalex publication_date 2019/10/28 · openalex created_date 2019/11/08 · arxiv created 2020/10/19 · arxiv updated 2020/10/21 · openalex updated_date 2026/07/28

Abstract

Generative Adversarial Networks (GANs) are a powerful class of generative models in the deep learning community. Current practice on large-scale GAN training utilizes large models and distributed large-batch training strategies, and is implemented on deep learning frameworks (e.g., TensorFlow, PyTorch, etc.) designed in a centralized manner. In the centralized network topology, every worker needs to either directly communicate with the central node or indirectly communicate with all other workers in every iteration. However, when the network bandwidth is low or network latency is high, the performance would be significantly degraded. Despite recent progress on decentralized algorithms for training deep neural networks, it remains unclear whether it is possible to train GANs in a decentralized manner. The main difficulty lies at handling the nonconvex-nonconcave min-max optimization and the decentralized communication simultaneously. In this paper, we address this difficulty by designing the first gradient-based decentralized parallel algorithm which allows workers to have multiple rounds of communications in one iteration and to update the discriminator and generator simultaneously, and this design makes it amenable for the convergence analysis of the proposed decentralized algorithm. Theoretically, our proposed decentralized algorithm is able to solve a class of non-convex non-concave min-max problems with provable non-asymptotic convergence to first-order stationary point. Experimental results on GANs demonstrate the effectiveness of the proposed algorithm.

Citations

• Monotone Operators and the Proximal Point Algorithm
• DSA: Decentralized Double Stochastic Averaging Gradient Algorithm
• Optimal algorithms for smooth and strongly convex distributed optimization in networks
• TAC-GAN - Text Conditioned Auxiliary Classifier Generative Adversarial Network
• An Online Learning Approach to Generative Adversarial Networks
• Certifying Some Distributional Robustness with Principled Adversarial Training
• Spectral Normalization for Generative Adversarial Networks
• A Variational Inequality Perspective on Generative Adversarial Networks
• An Alternative View: When Does SGD Escape Local Minima?
• D²: Decentralized Training over Decentralized Data
• Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Accelerated Decentralized Optimization with Local Updates for Smooth and Strongly Convex Objectives
• Solving Non-Convex Non-Concave Min-Max Games Under Polyak-Łojasiewicz Condition
• On Finding Local Nash Equilibria (and Only Local Nash Equilibria) in Zero-Sum Games
• Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication
• Reducing Noise in GAN Training with Variance Reduced Extragradient
• On the Linear Speedup Analysis of Communication Efficient Momentum SGD\n for Distributed Non-Convex Optimization
• MATCHA: Speeding Up Decentralized SGD via Matching Decomposition Sampling
• On Gradient Descent Ascent for Nonconvex-Concave Minimax Problems
• Adversarial Learning for Neural Dialogue Generation
• Variance-Reduced Decentralized Stochastic Optimization with Gradient Tracking--Part I: GT-SAGA
• SPARQ-SGD: Event-Triggered and Compressed Communication in Decentralized Stochastic Optimization
• Towards Better Understanding of Adaptive Gradient Algorithms in Generative Adversarial Nets

Cited by

• An Optimistic Gradient Tracking Method for Distributed Minimax Optimization
• Decentralized Min-Max Optimization with Gradient Tracking

Related

• A New Distributed Method for Training Generative Adversarial Networks
• A Distributed Training Algorithm of Generative Adversarial Networks with Quantized Gradients
• From promise to practice: realizing high-performance decentralized training
• Parallel/distributed implementation of cellular training for generative adversarial neural networks
• Decentralized Learning of Generative Adversarial Networks from Non-iid Data