Improving Deep Tabular Learning

2025/09/19 by Sivan Sarafian, Sarafian, Sivan, Yehudit Aperstein +1
Computer Science · #FOS: Computer and information sciences #Intelligent Tutoring Systems and Adaptive Learning #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2509.16354

openalex publication_date 2025/09/19 · openalex created_date 2025/10/16 · openalex updated_date 2026/07/28

Abstract

Tabular data remain a dominant form of real-world information but pose persistent challenges for deep learning due to heterogeneous feature types, lack of natural structure, and limited label-preserving augmentations. As a result, ensemble models based on decision trees continue to dominate benchmark leaderboards. In this work, we introduce RuleNet, a transformer-based architecture specifically designed for deep tabular learning. RuleNet incorporates learnable rule embeddings in a decoder, a piecewise linear quantile projection for numerical features, and feature masking ensembles for robustness and uncertainty estimation. Evaluated on eight benchmark datasets, RuleNet matches or surpasses state-of-the-art tree-based methods in most cases, while remaining computationally efficient, offering a practical neural alternative for tabular prediction tasks.

Citations

• TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023
• UniTabE: A Universal Pretraining Protocol for Tabular Foundation Model in Data Science
• A Survey on Evaluation of Large Language Models
• A Survey on Evaluation of Large Language Models
• When Do Neural Nets Outperform Boosted Trees on Tabular Data?
• TabLLM: Few-shot Classification of Tabular Data with Large Language Models
• Revisiting Pretraining Objectives for Tabular Deep Learning
• On Embeddings for Numerical Features in Tabular Deep Learning
• On the Opportunities and Risks of Foundation Models
• Revisiting Deep Learning Models for Tabular Data
• Tabular Data: Deep Learning is Not All You Need
• SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training
• TabTransformer: Tabular Data Modeling Using Contextual Embeddings
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data
• TabNet: Attentive Interpretable Tabular Learning
• Optuna: A Next-generation Hyperparameter Optimization Framework
• Tune: A Research Platform for Distributed Model Selection and Training
• Attention Is All You Need
• Gaussian Error Linear Units (GELUs)
• Greedy function approximation: A gradient boosting machine.
• Long Short-Term Memory
• Sparse spatial autoregressions

Related

• Revisiting Deep Learning Models for Tabular Data
• TabNet: Attentive Interpretable Tabular Learning
• Deep Learning within Tabular Data: Foundations, Challenges, Advances and Future Directions
• A Survey on Deep Tabular Learning
• Simple Modifications to Improve Tabular Neural Networks