Well-tuned Simple Nets Excel on Tabular Datasets

2021/06/21 by Arlind Kadra, Kadra, Arlind, Marius Lindauer +5 · 1 voice · 13 citations
Computer Science · #Advanced Neural Network Applications #Image Enhancement Techniques #Machine Learning and Data Classification #cs.LG

paper · pdf · doi:10.48550/arxiv.2106.11189

openalex publication_date 2021/06/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Tabular datasets are the last "unconquered castle" for deep learning, with traditional ML methods like Gradient-Boosted Decision Trees still performing strongly even against recent specialized neural architectures. In this paper, we hypothesize that the key to boosting the performance of neural networks lies in rethinking the joint and simultaneous application of a large set of modern regularization techniques. As a result, we propose regularizing plain Multilayer Perceptron (MLP) networks by searching for the optimal combination/cocktail of 13 regularization techniques for each dataset using a joint optimization over the decision on which regularizers to apply and their subsidiary hyperparameters. We empirically assess the impact of these regularization cocktails for MLPs in a large-scale empirical study comprising 40 tabular datasets and demonstrate that (i) well-regularized plain MLPs significantly outperform recent state-of-the-art specialized neural network architectures, and (ii) they even outperform strong traditional ML methods, such as XGBoost.

Cited by

• A Model of Causal Explanation on Neural Networks for Tabular Data
• Multiclass Graph-Based Large Margin Classifiers: Unified Approach for Support Vectors and Neural Networks
• AdaCap: An Adaptive Contrastive Approach for Small-Data Neural Networks
• Boosting Predictive Performance on Tabular Data through Data Augmentation with Latent-Space Flow-Based Diffusion
• iLTM: Integrated Large Tabular Model
• Tab-PET: Graph-Based Positional Encodings for Tabular Transformers
• Comparison of generalised additive models and neural networks in applications: A systematic review
• SCALAR: Self-Calibrating Adaptive Latent Attention Representation Learning
• Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Models
• IMLP: An Energy-Efficient Continual Learning Method for Tabular Data Streams
• Multi-level Diagnosis and Evaluation for Robust Tabular Feature Engineering with Large Language Models
• Probabilistic Pretraining for Neural Regression
• LLM Empowered Prototype Learning for Zero and Few-Shot Tasks on Tabular Data

Discussions

• Regularization is all you need: simple neural nets can excel on tabular data [hn, 216 points, 93 comments]

Related

• Regularization Learning Networks: Deep Learning for Tabular Datasets
• TANGOS: Regularizing Tabular Neural Networks through Gradient Orthogonalization and Specialization
• GCondNet: A Novel Method for Improving Neural Networks on Small High-Dimensional Tabular Data
• Tabular Data: Is Deep Learning all you need?
• When Do Neural Nets Outperform Boosted Trees on Tabular Data?