On the Principle of Least Symmetry Breaking in Shallow ReLU Models

2019/12/26 by Arjevani, Yossi, Field, Michael
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.1912.11939

Abstract

We consider the optimization problem associated with fitting two-layer ReLU networks with respect to the squared loss, where labels are assumed to be generated by a target network. Focusing first on standard Gaussian inputs, we show that the structure of spurious local minima detected by stochastic gradient descent (SGD) is, in a well-defined sense, the least loss of symmetry with respect to the target weights. A closer look at the analysis indicates that this principle of least symmetry breaking may apply to a broader range of settings. Motivated by this, we conduct a series of experiments which corroborate this hypothesis for different classes of non-isotropic non-product distributions, smooth activation functions and networks with a few layers.

Related

• Annihilation of Spurious Minima in Two-Layer ReLU Networks
• Analytic Characterization of the Hessian in Shallow ReLU Models: A Tale of Symmetry
• Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry II
• Hidden Minima in Two-Layer ReLU Networks
• The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networks