On the relationship between multivariate splines and infinitely-wide neural networks

2023/02/07 by Francis Bach, Bach, Francis · 1 citation
Computer Science · Engineering · Medicine · #Advanced Numerical Analysis Techniques #FOS: Computer and information sciences #FOS: Mathematics #Image Processing and 3D Reconstruction #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Radiomics and Machine Learning in Medical Imaging #Statistics Theory (math.ST)

paper · pdf · doi:10.48550/arxiv.2302.03459

openalex publication_date 2023/02/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

We consider multivariate splines and show that they have a random feature expansion as infinitely wide neural networks with one-hidden layer and a homogeneous activation function which is the power of the rectified linear unit. We show that the associated function space is a Sobolev space on a Euclidean ball, with an explicit bound on the norms of derivatives. This link provides a new random feature expansion for multivariate splines that allow efficient algorithms. This random feature expansion is numerically better behaved than usual random Fourier features, both in theory and practice. In particular, in dimension one, we compare the associated leverage scores to compare the two random expansions and show a better scaling for the neural network expansion.

Cited by

• A Convex Loss Function for Set Prediction with Optimal Trade-offs Between Size and Conditional Coverage

Related

• A comparison of deep networks with ReLU activation function and linear spline-type methods
• Multi-layer random features and the approximation power of neural networks
• Neural Splines: Fitting 3D Surfaces with Infinitely-Wide Neural Networks
• On the Approximation Properties of Random ReLU Features
• Points of non-linearity of functions generated by random neural networks