2021/08/12 by Steffen Dereich, Dereich, S., Sebastian Kassing +1
Computer Science · Engineering · Neuroscience · #26B40 #Advanced Memory and Neural Computing #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Neural Networks and Applications #Neural dynamics and brain function #Primary 68T05 #Secondary 68T07 #Statistics Theory (math.ST)
paper · pdf · doi:10.48550/arxiv.2108.05643
openalex publication_date 2021/08/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01
The realization function of a shallow ReLU network is a continuous and piecewise affine function f:\mathbb Rd→ \mathbb R, where the domain \mathbb Rd is partitioned by a set of n hyperplanes into cells on which f is affine. We show that the minimal representation for f uses either n, n+1 or n+2 neurons and we characterize each of the three cases. In the particular case, where the input layer is one-dimensional, minimal representations always use at most n+1 neurons but in all higher dimensional settings there are functions for which n+2 neurons are needed. Then we show that the set of minimal networks representing f forms a C^∞-submanifold M and we derive the dimension and the number of connected components of M. Additionally, we give a criterion for the hyperplanes that guarantees that all continuous, piecewise affine functions are realization functions of appropriate ReLU networks.