Geometric structure of Deep Learning networks and construction of global \mathcal L² minimizers

2023/09/19 by Thomas Y. Chen, Chen, Thomas, Patricia Muñoz Ewald +1 · 1 citation
Computer Science · Engineering · #57R70 #62M45 #Advanced Numerical Analysis Techniques #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #FOS: Mathematics #FOS: Physical sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Mathematical Physics (math-ph) #Medical Image Segmentation Techniques #Optimization and Control (math.OC) #Topology Optimization in Engineering

paper · pdf · doi:10.48550/arxiv.2309.10639

openalex publication_date 2023/09/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we explicitly determine local and global minimizers of the L² cost function in underparametrized Deep Learning (DL) networks; our main goal is to shed light on their geometric structure and properties. We accomplish this by a direct construction, without invoking the gradient descent flow at any point of this work. We specifically consider L hidden layers, a ReLU ramp activation function, an L² Schatten class (or Hilbert-Schmidt) cost function, input and output spaces ℝ^Q with equal dimension Q≥1, and hidden layers also defined on ℝ^Q; the training inputs are assumed to be sufficiently clustered. The training input size N can be arbitrarily large - thus, we are considering the underparametrized regime. More general settings are left to future work. We construct an explicit family of minimizers for the global minimum of the cost function in the case L≥ Q, which we show to be degenerate. Moreover, we determine a set of 2^Q-1 distinct degenerate local minima of the cost function. In the context presented here, the concatenation of hidden layers of the DL network is reinterpreted as a recursive application of a \em truncation map which "curates" the training inputs by minimizing their noise to signal ratio.

Cited by

• An entropy formula for the Deep Linear Network

Related

• Geometric structure of shallow neural networks and constructive \mathcal L² cost minimization
• Global L² minimization at uniform exponential rate via geometrically adapted gradient descent in Deep Learning
• On non-approximability of zero loss global \mathcal L² minimizers by gradient descent in Deep Learning
• Zero loss guarantees and explicit minimizers for generic overparametrized Deep Learning networks
• The loss surface of deep and wide neural networks