Geometry and Optimization of Shallow Polynomial Networks

2025/01/10 by Arjevani, Yossi, Bruna, Joan, Kileel, Joe +2 · 1 citation
#14P10 #62R01 #68T07 #90C23 #Algebraic Geometry (math.AG) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2501.06074

Abstract

We study shallow neural networks with monomial activations and output dimension one. The function space for these models can be identified with a set of symmetric tensors with bounded rank. We describe general features of these networks, focusing on the relationship between width and optimization. We then consider teacher-student problems, which can be viewed as problems of low-rank tensor approximation with respect to non-standard inner products that are induced by the data distribution. In this setting, we introduce a teacher-metric data discriminant which encodes the qualitative behavior of the optimization as a function of the training data distribution. Finally, we focus on networks with quadratic activations, presenting an in-depth analysis of the optimization landscape. In particular, we present a variation of the Eckart-Young Theorem characterizing all critical points and their Hessian signatures for teacher-student problems with quadratic networks and Gaussian training data.

Cited by

• Critical Points of Degenerate Metrics on Algebraic Varieties: A Tale of Overparametrization

Related

• Stationary Points of Shallow Neural Networks with Quadratic Activation\n Function
• Landscape analysis for shallow neural networks: Complete classification of critical points for cubic activation and affine target functions
• On the Geometry and Optimization of Polynomial Convolutional Networks
• On the Expressive Power of Deep Polynomial Neural Networks
• Stationary Points of a Shallow Neural Network with Quadratic Activations and the Global Optimality of the Gradient Descent Algorithm