2020/03/03 by Hadi Daneshmand, Daneshmand, Hadi, Jonas Köhler +9 · 9 citations
Computer Science · Engineering · Mathematics · #Algorithm #Artificial intelligence #Artificial neural network #Combinatorics #Computer science #Deep neural networks #FOS: Computer and information sciences #Face and Expression Recognition #Initialization #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine learning #Markov chain #Mathematics #Normalization (sociology) #Rank (graph theory) #Residual #Robustness (evolution) #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques #Theoretical computer science #cs.LG #stat.ML
paper · pdf · doi:10.48550/arxiv.2003.01652
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2020/03/03 · arxiv created 2020/06/11 · arxiv updated 2020/06/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Randomly initialized neural networks are known to become harder to train with increasing depth, unless architectural enhancements like residual connections and batch normalization are used. We here investigate this phenomenon by revisiting the connection between random initialization in deep networks and spectral instabilities in products of random matrices. Given the rich literature on random matrices, it is not surprising to find that the rank of the intermediate representations in unnormalized networks collapses quickly with depth. In this work we highlight the fact that batch normalization is an effective strategy to avoid rank collapse for both linear and ReLU networks. Leveraging tools from Markov chain theory, we derive a meaningful lower rank bound in deep linear networks. Empirically, we also demonstrate that this rank robustness generalizes to ReLU nets. Finally, we conduct an extensive set of experiments on real-world data sets, which confirm that rank stability is indeed a crucial condition for training modern-day deep neural architectures.