Periodic Spectral Ergodicity: A Complexity Measure for Deep Neural\n Networks and Neural Architecture Search

2019/11/09 by Mehmet Lutfi Süzen, Süzen, Mehmet, Joan J. Cerdà +4 · 1 citation
Biochemistry, Genetics and Molecular Biology · Computer Science · Materials Science · Physics and Astronomy · #68-02 #Disordered Systems and Neural Networks (cond-mat.dis-nn) #FOS: Computer and information sciences #FOS: Physical sciences #Fractal and DNA sequence analysis #G.3 #I.2.6 #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning in Materials Science #Neural Networks and Applications #Statistical Mechanics (cond-mat.stat-mech) #Statistical Mechanics and Entropy

paper · pdf · doi:10.48550/arxiv.1911.07831

openalex publication_date 2019/11/09 · openalex created_date 2022/07/26 · openalex updated_date 2026/07/28

Abstract

Establishing associations between the structure and the generalisation\nability of deep neural networks (DNNs) is a challenging task in modern machine\nlearning. Producing solutions to this challenge will bring progress both in the\ntheoretical understanding of DNNs and in building new architectures\nefficiently. In this work, we address this challenge by developing a new\ncomplexity measure based on the concept of Periodic Spectral Ergodicity (PSE)\noriginating from quantum statistical mechanics. Based on this measure a\ntechnique is devised to quantify the complexity of deep neural networks from\nthe learned weights and traversing the network connectivity in a sequential\nmanner, hence the term cascading PSE (cPSE), as an empirical complexity\nmeasure. This measure will capture both topological and internal neural\nprocessing complexity simultaneously. Because of this cascading approach, i.e.,\na symmetric divergence of PSE on the consecutive layers, it is possible to use\nthis measure for Neural Architecture Search (NAS). We demonstrate the\nusefulness of this measure in practice on two sets of vision models, ResNet and\nVGG, and sketch the computation of cPSE for more complex network structures.\n

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Learning in Neural Networks: An Overview
• Detecting Statistical Interactions from Neural Network Weights
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• The Emergence of Spectral Universality in Deep Networks
• Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning
• HARK Side of Deep Learning -- From Grad Student Descent to Automated\n Machine Learning
• PyTorch: An Imperative Style, High-Performance Deep Learning Library

Cited by

• Ergodicity detection algorithms: Scaling of ergodicity in random symbolic dynamics

Related

• Spectral complexity of deep neural networks
• Complexity for deep neural networks and other characteristics of deep feature representations
• Deep neural networks from the perspective of ergodic theory
• Equivalence in Deep Neural Networks via Conjugate Matrix Ensembles
• On Characterizing the Capacity of Neural Networks using Algebraic Topology