Convergence of neural networks to Gaussian mixture distribution

2022/04/26 by Yasuhiko Asao, Asao, Yasuhiko, Ryotaro Sakamoto +3 · 1 citation
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Neural Networks and Applications #Probability (math.PR) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2204.12100

openalex publication_date 2022/04/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We give a proof that, under relatively mild conditions, fully-connected feed-forward deep random neural networks converge to a Gaussian mixture distribution as only the width of the last hidden layer goes to infinity. We conducted experiments for a simple model which supports our result. Moreover, it gives a detailed description of the convergence, namely, the growth of the last hidden layer gets the distribution closer to the Gaussian mixture, and the other layer successively get the Gaussian mixture closer to the normal distribution.

Cited by

• Learning Representations on the Unit Sphere: Investigating Angular Gaussian and von Mises-Fisher Distributions for Online Continual Learning

Related

• Random Neural Networks in the Infinite Width Limit as Gaussian Processes
• Quantitative Gaussian Approximation of Randomly Initialized Deep Neural Networks
• Proportional infinite-width infinite-depth limit for deep linear neural networks
• Gaussian Process Behaviour in Wide Deep Neural Networks
• Wide Deep Neural Networks with Gaussian Weights are Very Close to Gaussian Processes