Muddling Label Regularization: Deep Learning for Tabular Datasets

2021/06/08 by Karim Lounici, Lounici, Karim, Katia Méziani +3
Computer Science · #68T07 #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Image Retrieval and Classification Techniques #Machine Learning (cs.LG) #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.2106.04462

openalex publication_date 2021/06/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Deep Learning (DL) is considered the state-of-the-art in computer vision, speech recognition and natural language processing. Until recently, it was also widely accepted that DL is irrelevant for learning tasks on tabular data, especially in the small sample regime where ensemble methods are acknowledged as the gold standard. We present a new end-to-end differentiable method to train a standard FFNN. Our method, Muddling labels for Regularization (MLR), penalizes memorization through the generation of uninformative labels and the application of a differentiable close-form regularization scheme on the last hidden layer during training. MLR outperforms classical NN and the gold standard (GBDT, RF) for regression and classification tasks on several datasets from the UCI database and Kaggle covering a large range of sample sizes and feature to sample ratios. Researchers and practitioners can use MLR on its own as an off-the-shelf \DL solution or integrate it into the most advanced ML pipelines.

Citations

• Stochastic gradient boosting
• Multivariate Adaptive Regression Splines
• Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups
• ImageNet classification with deep convolutional neural networks
• Deep Learning over Multi-field Categorical Data: A Case Study on User\n Response Prediction
• Entity Embeddings of Categorical Variables
• Forward Thinking: Building Deep Random Forests
• Deep Neural Decision Trees
• Understanding deep learning requires rethinking generalization
• Deep Interest Network for Click-Through Rate Prediction
• Learning to Explain: An Information-Theoretic Perspective on Model Interpretation
• Fastai: A Layered API for Deep Learning
• Muddling Labels for Regularization, a novel approach to generalization

Related

• Regularization Learning Networks: Deep Learning for Tabular Datasets
• Adaptive Regularization of Labels
• Regularizing Deep Networks by Modeling and Predicting Label Structure
• Do we need Label Regularization to Fine-tune Pre-trained Language Models?
• Well-tuned Simple Nets Excel on Tabular Datasets