2023/05/28 by Amir Joudaki, Joudaki, Amir, Hadi Daneshmand +3 · 2 citations
Computer Science · Neuroscience · #Artificial Intelligence (cs.AI) #Blind Source Separation Techniques #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications #Visual perception and processing mechanisms
paper · pdf · doi:10.48550/arxiv.2305.18399
openalex publication_date 2023/05/28 · openalex created_date 2023/06/01 · openalex updated_date 2026/08/01
In this paper, we explore the structure of the penultimate Gram matrix in deep neural networks, which contains the pairwise inner products of outputs corresponding to a batch of inputs. In several architectures it has been observed that this Gram matrix becomes degenerate with depth at initialization, which dramatically slows training. Normalization layers, such as batch or layer normalization, play a pivotal role in preventing the rank collapse issue. Despite promising advances, the existing theoretical results do not extend to layer normalization, which is widely used in transformers, and can not quantitatively characterize the role of non-linear activations. To bridge this gap, we prove that layer normalization, in conjunction with activation layers, biases the Gram matrix of a multilayer perceptron towards the identity matrix at an exponential rate with depth at initialization. We quantify this rate using the Hermite expansion of the activation function.