Towards a mathematical framework to inform Neural Network modelling via Polynomial Regression

2021/02/07 by Pablo Morala, Jenny Cifuentes, Jenny Alexandra Cifuentes +3 · 1 voice · 1 citation
Computer Science · Mathematics · Physics and Astronomy · #Machine Learning and Data Classification #Model Reduction and Neural Networks #Neural Networks and Applications #cs.LG #stat.ML

paper · pdf · doi:10.1016/j.neunet.2021.04.036

published as Neural Networks 142 (2021), 57-72 · 39 pages, 15 figures

arxiv created 2021/02/07 · arxiv published 2021/02/07 · openalex publication_date 2021/04/30 · crossref created 2021/04/30 · arxiv updated 2021/05/11 · crossref issued 2021/10/01 · crossref published 2021/10/01 · crossref published-print 2021/10/01 · openalex created_date 2025/10/10 · crossref deposited 2025/10/12 · crossref indexed 2026/01/08 · openalex updated_date 2026/07/28

Abstract

Even when neural networks are widely used in a large number of applications, they are still considered as black boxes and present some difficulties for dimensioning or evaluating their prediction error. This has led to an increasing interest in the overlapping area between neural networks and more traditional statistical methods, which can help overcome those problems. In this article, a mathematical framework relating neural networks and polynomial regression is explored by building an explicit expression for the coefficients of a polynomial regression from the weights of a given neural network, using a Taylor expansion approach. This is achieved for single hidden layer neural networks in regression problems. The validity of the proposed method depends on different factors like the distribution of the synaptic potentials or the chosen activation function. The performance of this method is empirically tested via simulation of synthetic data generated from polynomials to train neural networks with different structures and hyperparameters, showing that almost identical predictions can be obtained when certain conditions are met. Lastly, when learning from polynomial generated data, the proposed method produces polynomials that approximate correctly the data locally.

Citations

• Lower Upper Bound Estimation Method for Construction of Neural Network-Based Prediction Intervals
• Multilayer feedforward networks are universal approximators
• Comprehensive Review of Neural Network-Based Prediction Intervals and New Advances
• Opening the Black Box of Deep Neural Networks via Information
• Understanding Black-box Predictions via Influence Functions
• Polynomial Regression As an Alternative to Neural Nets
• A Survey Of Methods For Explaining Black Box Models
• On a Method to Measure Supervised Multiclass Model’s Interpretability: Application to Degradation Diagnosis (Short Paper)

Cited by

• Comprehensive review of optimization and surrogate models for agricultural water resources and reservoir water management

Discussions

• Our first steps "Towards a mathematical framework to inform Neural Network modelling via Polynomial Regression" are on arXiv https://arxiv.org/abs/2102.03865 (kudos to @moralapablo), building on https [bsky, 0 points, 0 comments]

Related

• Polynomial Regression As an Alternative to Neural Nets
• Ladder Polynomial Neural Networks
• AMITE: A Novel Polynomial Expansion for Analyzing Neural Network Nonlinearities
• Bayesian polynomial neural networks and polynomial neural ordinary differential equations
• Neural Networks and Polynomial Regression. Demystifying the Overparametrization Phenomena