2018/06/05 by Peter Hinz, Hinz, Peter, Sara van de Geer +1 · 1 citation
Computer Science · Materials Science · Physics and Astronomy · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning in Materials Science #Model Reduction and Neural Networks #Neural Networks and Applications
paper · pdf · doi:10.48550/arxiv.1806.01918
openalex publication_date 2018/06/05 · openalex created_date 2022/10/03 · openalex updated_date 2026/07/28
We present a framework to derive upper bounds on the number of regions that\nfeed-forward neural networks with ReLU activation functions are affine linear\non. It is based on an inductive analysis that keeps track of the number of such\nregions per dimensionality of their images within the layers. More precisely,\nthe information about the number regions per dimensionality is pushed through\nthe layers starting with one region of the input dimension of the neural\nnetwork and using a recursion based on an analysis of how many regions per\noutput dimensionality a subsequent layer with a certain width can induce on an\ninput region with a given dimensionality. The final bound on the number of\nregions depends on the number and widths of the layers of the neural network\nand on some additional parameters that were used for the recursion. It is\nstated in terms of the L1-norm of the last column of a product of matrices\nand provides a unifying treatment of several previously known bounds: Depending\non the choice of the recursion parameters that determine these matrices, it is\npossible to obtain the bounds from Mont 'ufar (2014), (2017) and Serra et.\nal. (2017) as special cases. For the latter, which is the strongest of these\nbounds, the formulation in terms of matrices provides new insight. In\nparticular, by using explicit formulas for a Jordan-like decomposition of the\ninvolved matrices, we achieve new tighter results for the asymptotic setting,\nwhere the number of layers of the same fixed width tends to infinity.\n