Exploring Weight Symmetry in Deep Neural Networks

2018/12/28 by Xu Shell Hu, Sergey Zagoruyko, Hu, Xu Shell +3 · 2 citations
Computer Science · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.1812.11027

openalex publication_date 2018/12/28 · openalex created_date 2019/08/13 · openalex updated_date 2026/07/28

Abstract

We propose to impose symmetry in neural network parameters to improve parameter usage and make use of dedicated convolution and matrix multiplication routines. Due to significant reduction in the number of parameters as a result of the symmetry constraints, one would expect a dramatic drop in accuracy. Surprisingly, we show that this is not the case, and, depending on network size, symmetry can have little or no negative effect on network accuracy, especially in deep overparameterized networks. We propose several ways to impose local symmetry in recurrent and convolutional neural networks, and show that our symmetry parameterizations satisfy universal approximation property for single hidden layer networks. We extensively evaluate these parameterizations on CIFAR, ImageNet and language modeling datasets, showing significant benefits from the use of symmetry. For instance, our ResNet-101 with channel-wise symmetry has almost 25% less parameters and only 0.2% accuracy loss on ImageNet. Code for our experiments is available at https://github.com/hushell/deep-symmetry

Citations

• Recurrent Neural Network Regularization
• ImageNet classification with deep convolutional neural networks
• Densely Connected Convolutional Networks
• Highway and Residual Networks learn Unrolled Iterative Estimation
• ShaResNet: reducing residual network parameter number by sharing weights
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Theoretical Properties for Neural Networks with Weight Matrices of Low Displacement Rank

Cited by

• Symmetry and Generalisation in Neural Approximations of Renormalisation Transformations
• Subdifferentiation with symmetry

Related

• Symmetry-invariant optimization in deep networks
• Symmetry in Neural Network Parameter Spaces
• ResNet and Asymmetric ResNet Weights, for "The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof"
• Symmetry Induces Structure and Constraint of Learning
• Understanding symmetries in deep networks