Large Deviations of Gaussian Neural Networks with ReLU activation

2024/05/27 by Vogel, Quirin
#60F10 #68T07 #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Probability (math.PR)

paper · doi:10.48550/arxiv.2405.16958

Abstract

We prove a large deviation principle for deep neural networks with Gaussian weights and at most linearly growing activation functions, such as ReLU. This generalises earlier work, in which bounded and continuous activation functions were considered. In practice, linearly growing activation functions such as ReLU are most commonly used. We furthermore simplify previous expressions for the rate function and provide a power-series expansions for the ReLU case.

Related

• Large and moderate deviations for Gaussian neural networks
• Beyond ReLU: How Activations Affect Neural Kernels and Random Wide Networks
• N-ReLU: Zero-Mean Stochastic Extension of ReLU
• Wide Deep Neural Networks with Gaussian Weights are Very Close to Gaussian Processes
• On the asymptotics of wide networks with polynomial activations