2018/04/04 by Massimo Fornasier, Fornasier, Massimo, Jan Vybíral +3
Computer Science · #Advanced Neural Network Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Neural Networks and Applications
paper · pdf · doi:10.48550/arxiv.1804.01592
openalex publication_date 2018/04/04 · openalex created_date 2022/08/22 · openalex updated_date 2026/08/01
We address the structure identification and the uniform approximation of sums\nof ridge functions f(x)=\∑i=1m gi(ai\⋅ x) on mathbb Rd,\nrepresenting a general form of a shallow feed-forward neural network, from a\nsmall number of query samples. Higher order differentiation, as used in our\nconstructive approximations, of sums of ridge functions or of their\ncompositions, as in deeper neural network, yields a natural connection between\nneural network weight identification and tensor product decomposition\nidentification. In the case of the shallowest feed-forward neural network,\nsecond order differentiation and tensors of order two (i.e., matrices) suffice\nas we prove in this paper. We use two sampling schemes to perform approximate\ndifferentiation - active sampling, where the sampling points are universal,\nactively, and randomly designed, and passive sampling, where sampling points\nwere preselected at random from a distribution with known density. Based on\nmultiple gathered approximated first and second order differentials, our\ngeneral approximation strategy is developed as a sequence of algorithms to\nperform individual sub-tasks. We first perform an active subspace search by\napproximating the span of the weight vectors a1,\…,am. Then we use a\nstraightforward substitution, which reduces the dimensionality of the problem\nfrom d to m. The core of the construction is then the stable and efficient\napproximation of weights expressed in terms of rank-1 matrices ai \⊗\nai, realized by formulating their individual identification as a suitable\nnonlinear program. We prove the successful identification by this program of\nweight vectors being close to orthonormal and we also show how we can\ncostructively reduce to this case by a whitening procedure, without loss of any\ngenerality.\n