On Symmetry and Initialization for Neural Networks

2019/07/01 by Ido Nachum, Nachum, Ido, Amir Yehudayoff +1 · 1 citation
Computer Science · Mathematics · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Neural Networks and Applications #Stochastic Gradient Optimization Techniques #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1907.00560

arxiv created 2019/07/01 · openalex publication_date 2019/07/01 · arxiv updated 2019/07/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This work provides an additional step in the theoretical understanding of neural networks. We consider neural networks with one hidden layer and show that when learning symmetric functions, one can choose initial conditions so that standard SGD training efficiently produces generalization guarantees. We empirically verify this and show that this does not hold when the initial conditions are chosen at random. The proof of convergence investigates the interaction between the two layers of the network. Our results highlight the importance of using symmetry in the design of neural networks.

Citations

• The perceptron: A probabilistic model for information storage and organization in the brain.
• No bad local minima: Data independent training error guarantees for multilayer neural networks
• Large-Margin Softmax Loss for Convolutional Neural Networks
• On the Complexity of Learning Neural Networks
• Learning Overparameterized Neural Networks via Stochastic Gradient\n Descent on Structured Data
• A Convergence Theory for Deep Learning via Over-Parameterization
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks

Cited by

• Symmetry and Generalisation in Neural Approximations of Renormalisation Transformations

Related

• A Functional Perspective on Learning Symmetric Functions with Neural Networks
• Symmetry-via-Duality: Invariant Neural Network Densities from Parameter-Space Correlators
• On the non-universality of deep learning: quantifying the cost of symmetry
• Noether: The More Things Change, the More Stay the Same
• The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks