Revisiting Chebyshev Polynomial and Anisotropic RBF Models for Tabular Regression

2026/02/25 by Luciano Gerber, Huw Lloyd · 1 voice
Computer Science · Mathematics · #Benchmark (surveying) #Benchmarking #Chebyshev polynomials #Differentiable function #Gaussian Processes and Bayesian Inference #Polynomial #Regression #Sensitivity (control systems) #Stochastic Gradient Optimization Techniques #Tensor decomposition and applications #Tree (set theory) #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.2602.22422

openalex publication_date 2026/02/25 · arxiv published 2026/02/25 · openalex created_date 2026/02/28 · arxiv updated 2026/06/14 · openalex updated_date 2026/07/28

Abstract

Smooth-basis models such as Chebyshev polynomial regressors and radial basis function (RBF) networks are well established in numerical analysis. Their continuously differentiable prediction surfaces suit surrogate optimisation, sensitivity analysis, and other settings where the response varies gradually with inputs. Despite these properties, smooth models seldom appear in tabular regression, where tree ensembles dominate. We ask whether they can compete, benchmarking models across 55 regression datasets organised by application domain. We develop an anisotropic RBF network with data-driven centre placement and gradient-based width optimisation, a ridge-regularised Chebyshev polynomial regressor, and a smooth-tree hybrid (Chebyshev model tree); all three are released as scikit-learn-compatible packages. We benchmark these against tree ensembles, a pre-trained transformer, and standard baselines, evaluating accuracy alongside generalisation behaviour. The transformer ranks first on accuracy across a majority of datasets, but its GPU dependence, inference latency, and dataset-size limits constrain deployment in the CPU-based settings common across applied science and industry. Among CPU-viable models, smooth models and tree ensembles are statistically tied on accuracy, but the former tend to exhibit tighter generalisation gaps. We recommend routinely including smooth-basis models in the candidate pool, particularly when downstream use benefits from tighter generalisation and gradually varying predictions.

Citations

• TabICL: A Tabular Foundation Model for In-Context Learning on Large Data
• KAN: Kolmogorov-Arnold Networks
• When Do Neural Nets Outperform Boosted Trees on Tabular Data?
• Revisiting Deep Learning Models for Tabular Data
• Tabular Data: Deep Learning is Not All You Need
• PMLB v1.0: An open source dataset collection for benchmarking machine learning methods
• Array Programming with NumPy
• Neural Additive Models: Interpretable Machine Learning with Neural Nets
• InterpretML: A Unified Framework for Machine Learning Interpretability
• Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data
• TabNet: Attentive Interpretable Tabular Learning
• Optuna: A Next-generation Hyperparameter Optimization Framework
• Reconciling modern machine-learning practice and the classical bias–variance trade-off
• MoleculeNet: A Benchmark for Molecular Machine Learning
• XGBoost: A Scalable Tree Boosting System
• OpenML
• Scikit-learn: Machine Learning in Python
• Matplotlib: A 2D Graphics Environment
• Ridge Regression: Biased Estimation for Nonorthogonal Problems
• Bagging predictors
• A Limited Memory Algorithm for Bound Constrained Optimization
• Orthogonal least squares learning algorithm for radial basis function networks

Discussions

• Smooth models over XGBoost/Random Forest for tabular regression - does it really mean trading accuracy for smoothness? My colleague @huwlloyd.bsky.social and I tested it: 55 datasets, 8 models, nested [bsky, 0 points, 1 comments]

Related

• Bagged Polynomial Regression and Neural Networks
• Why do tree-based models still outperform deep learning on tabular data?
• TabICLv2: A better, faster, scalable, and open tabular foundation model
• xRFM: Accurate, scalable, and interpretable feature learning models for tabular data
• TabPFN: One Model to Rule Them All?