Stable behaviour of infinitely wide deep neural networks

2020/03/01 by Stefano Favaro, Sandra Fortini, Favaro, Stefano +3 · 1 citation
Computer Science · Mathematics · #Stochastic Gradient Optimization Techniques #Gaussian Processes and Bayesian Inference #Markov Chains and Monte Carlo Methods

paper · pdf · doi:10.48550/arxiv.2003.00394

Abstract

We consider fully connected feed-forward deep neural networks (NNs) where weights and biases are independent and identically distributed as symmetric centered stable distributions. Then, we show that the infinite wide limit of the NN, under suitable scaling on the weights, is a stochastic process whose finite-dimensional distributions are multivariate stable distributions. The limiting process is referred to as the stable process, and it generalizes the class of Gaussian processes recently obtained as infinite wide limits of NNs (Matthews at al., 2018b). Parameters of the stable process can be computed via an explicit recursion over the layers of the network. Our result contributes to the theory of fully connected feed-forward deep NNs, and it paves the way to expand recent lines of research that rely on Gaussian infinite wide limits.

Cited by

• Optimal High-probability Convergence of Nonlinear SGD under Heavy-tailed Noise via Symmetrization
• Avoiding Kernel Fixed Points: Computing with ELU and GELU Infinite Networks

Related

• Infinitely wide limits for deep Stable neural networks: sub-linear, linear and super-linear activation functions
• Deep Stable neural networks: large-width asymptotics and convergence rates
• α-Stable convergence of heavy-tailed infinitely-wide neural networks
• Steps Toward Deep Kernel Methods from Infinite Neural Networks
• Proportional infinite-width infinite-depth limit for deep linear neural networks