2020/08/06 by Alexander Cloninger, Cloninger, Alexander, Timo Klock +1 · 1 citation
Computer Science · Physics and Astronomy · #41A25 #41A46 #41A63 #62G05 #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Model Reduction and Neural Networks #Neural Networks and Applications #Statistics Theory (math.ST) #Stochastic Gradient Optimization Techniques
paper · pdf · doi:10.48550/arxiv.2008.02545
openalex publication_date 2020/08/06 · openalex created_date 2022/07/26 · openalex updated_date 2026/07/28
We study the approximation of two-layer compositions f(x) = g(\φ(x)) via\ndeep networks with ReLU activation, where \φ is a geometrically intuitive,\ndimensionality reducing feature map. We focus on two intuitive and practically\nrelevant choices for \φ: the projection onto a low-dimensional embedded\nsubmanifold and a distance to a collection of low-dimensional sets. We achieve\nnear optimal approximation rates, which depend only on the complexity of the\ndimensionality reducing map \φ rather than the ambient dimension. Since\n\φ encapsulates all nonlinear features that are material to the function\nf, this suggests that deep nets are faithful to an intrinsic dimension\ngoverned by f rather than the complexity of the domain of f. In particular,\nthe prevalent assumption of approximating functions on low-dimensional\nmanifolds can be significantly relaxed using functions of type f(x) =\ng(\φ(x)) with \φ representing an orthogonal projection onto the same\nmanifold.\n