2018/05/22 by Chuang Wang, Hong Hu, Wang, Chuang +3 · 3 citations
Computer Science · Engineering · Physics and Astronomy · #Disordered Systems and Neural Networks (cond-mat.dis-nn) #FOS: Computer and information sciences #FOS: Physical sciences #Generative Adversarial Networks and Image Synthesis #Information Theory (cs.IT) #Lattice Boltzmann Simulation Studies #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Model Reduction and Neural Networks
paper · pdf · doi:10.48550/arxiv.1805.08349
openalex publication_date 2018/05/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
We present a theoretical analysis of the training process for a single-layer GAN fed by high-dimensional input data. The training dynamics of the proposed model at both microscopic and macroscopic scales can be exactly analyzed in the high-dimensional limit. In particular, we prove that the macroscopic quantities measuring the quality of the training process converge to a deterministic process characterized by an ordinary differential equation (ODE), whereas the microscopic states containing all the detailed weights remain stochastic, whose dynamics can be described by a stochastic differential equation (SDE). This analysis provides a new perspective different from recent analyses in the limit of small learning rate, where the microscopic state is always considered deterministic, and the contribution of noise is ignored. From our analysis, we show that the level of the background noise is essential to the convergence of the training process: setting the noise level too strong leads to failure of feature recovery, whereas setting the noise too weak causes oscillation. Although this work focuses on a simple copy model of GAN, we believe the analysis methods and insights developed here would prove useful in the theoretical understanding of other variants of GANs with more advanced training algorithms.