2019/11/26 by Siavash Golkar, Golkar, Siavash
Computer Science · #Computational Physics and Python Applications #FOS: Biological sciences #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications #Neural and Evolutionary Computing (cs.NE) #Neurons and Cognition (q-bio.NC) #Time Series Analysis and Forecasting
paper · pdf · doi:10.48550/arxiv.1911.11691
openalex publication_date 2019/11/26 · openalex created_date 2019/12/05 · openalex updated_date 2026/07/28
Motivated by the flexibility of biological neural networks whose connectivity structure changes significantly during their lifetime, we introduce the Unstructured Recursive Network (URN) and demonstrate that it can exhibit similar flexibility during training via gradient descent. We show empirically that many of the different neural network structures commonly used in practice today (including fully connected, locally connected and residual networks of different depths and widths) can emerge dynamically from the same URN. These different structures can be derived using gradient descent on a single general loss function where the structure of the data and the relative strengths of various regulator terms determine the structure of the emergent network. We show that this loss function and the regulators arise naturally when considering the symmetries of the network as well as the geometric properties of the input data.