2020/06/16 by Huy Tuan Pham, Pham, Huy Tuan, Phan-Minh Nguyen +1
Computer Science · #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Neural Networks and Applications #Optimization and Control (math.OC) #Statistics Theory (math.ST) #Stochastic Gradient Optimization Techniques
paper · pdf · doi:10.48550/arxiv.2006.09355
openalex publication_date 2020/06/16 · openalex created_date 2022/07/26 · openalex updated_date 2026/07/28
In a recent work, we introduced a rigorous framework to describe the mean\nfield limit of the gradient-based learning dynamics of multilayer neural\nnetworks, based on the idea of a neuronal embedding. There we also proved a\nglobal convergence guarantee for three-layer (as well as two-layer) networks\nusing this framework.\n In this companion note, we point out that the insights in our previous work\ncan be readily extended to prove a global convergence guarantee for multilayer\nnetworks of any depths. Unlike our previous three-layer global convergence\nguarantee that assumes i.i.d. initializations, our present result applies to a\ntype of correlated initialization. This initialization allows to, at any finite\ntraining time, propagate a certain universal approximation property through the\ndepth of the neural network. To achieve this effect, we introduce a\nbidirectional diversity condition.\n