The global optimum of shallow neural network is attained by ridgelet transform

2018/05/19 by Sho Sonoda, Isao Ishikawa, Sonoda, Sho +11
Computer Science · #FOS: Computer and information sciences #Face and Expression Recognition #Image and Signal Denoising Methods #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.1805.07517

openalex publication_date 2018/05/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We prove that the global minimum of the backpropagation (BP) training problem of neural networks with an arbitrary nonlinear activation is given by the ridgelet transform. A series of computational experiments show that there exists an interesting similarity between the scatter plot of hidden parameters in a shallow neural network after the BP training and the spectrum of the ridgelet transform. By introducing a continuous model of neural networks, we reduce the training problem to a convex optimization in an infinite dimensional Hilbert space, and obtain the explicit expression of the global optimizer via the ridgelet transform.

Citations

• The Loss Surfaces of Multilayer Networks
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• No bad local minima: Data independent training error guarantees for\n multilayer neural networks
• Depth Creates No Bad Local Minima
• The loss surface of deep and wide neural networks
• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
• On the Equivalence between Kernel Quadrature Rules and Random Feature Expansions

Related

• Stationary Points of a Shallow Neural Network with Quadratic Activations and the Global Optimality of the Gradient Descent Algorithm
• Greedy Shallow Networks: An Approach for Constructing and Training Neural Networks
• Gradient Descent Finds Global Minima for Generalizable Deep Neural Networks of Practical Sizes
• Gradient Descent Finds Global Minima of Deep Neural Networks
• Theoretical insights into the optimization landscape of over-parameterized shallow neural networks