TabTransformer: Tabular Data Modeling Using Contextual Embeddings

2020/12/11 by Xin Huang, Ashish Khetan, Huang, Xin +5 · 99 citations
Computer Science · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #Explainable Artificial Intelligence (XAI) #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.2012.06678

7 pages, 5 figures

arxiv created 2020/12/11 · arxiv updated 2020/12/15

Abstract

We propose TabTransformer, a novel deep tabular data modeling architecture for supervised and semi-supervised learning. The TabTransformer is built upon self-attention based Transformers. The Transformer layers transform the embeddings of categorical features into robust contextual embeddings to achieve higher prediction accuracy. Through extensive experiments on fifteen publicly available datasets, we show that the TabTransformer outperforms the state-of-the-art deep learning methods for tabular data by at least 1.0% on mean AUC, and matches the performance of tree-based ensemble models. Furthermore, we demonstrate that the contextual embeddings learned from TabTransformer are highly robust against both missing and noisy data features, and provide better interpretability. Lastly, for the semi-supervised setting we develop an unsupervised pre-training procedure to learn data-driven contextual embeddings, resulting in an average 2.1% AUC lift over the state-of-the-art methods.

Cited by

• Calibrated Tree-Neural Fusion for Fine-Grained Vegetation Community Classification
• A Cross-lingual Comparison of Human and Classification Model Entrainment Behavior in Code-switched Speech Settings
• Tokenizing Numerical and Embedding Features for LLM RecSys
• Cyber Threat Detection Enabled by Quantum Computing
• Improving Cardiac Risk Prediction Using Data Generation Techniques
• Early Warning Index for Patient Deteriorations in Hospitals
• XNNTab -- Interpretable Neural Networks for Tabular Data using Sparse Autoencoders
• Teaching a Transformer to Think Like a Chemist: Predicting Nanocluster Stability
• IVGAE: Handling Incomplete Heterogeneous Data with a Variational Graph Autoencoder
• Light-Weight Benchmarks Reveal the Hidden Hardware Cost of Zero-Shot Tabular Foundation Models
• ABLE: Using Adversarial Pairs to Construct Local Models for Explaining Model Predictions
• Probabilistic Hash Embeddings for Online Learning of Categorical Features
• iLTM: Integrated Large Tabular Model
• FinTRec: Transformer Based Unified Contextual Ads Targeting and Personalization for Financial Applications
• Tab-PET: Graph-Based Positional Encodings for Tabular Transformers
• Uncertainty-Aware Multimodal Fusion for Oral Lesion Classification
• TransactionGPT
• KoTaP: A Panel Dataset for Corporate Tax Avoidance, Performance, and Governance in Korea
• What Does It Take to Detect an AI Agent? Minimal Feature Sets for Behavioral Detection under Browser Automation
• Forecasting Occupational Survivability of Rickshaw Pullers in a Changing Climate with Wearable Data
• MMbeddings: Parameter-Efficient, Low-Overfitting Probabilistic Embeddings Inspired by Nonlinear Mixed Models
• Strengthening LLMs for Tabular Prediction with Structural Priors
• FairContrast: Enhancing Fairness through Contrastive learning and Customized Augmenting Methods on Tabular Data
• ReSSFormer: A Recursive Sparse Structured Transformer for Scalable and Long-Context Reasoning
• The Role of Feature Interactions in Graph-based Tabular Deep Learning
• TabINR: An Implicit Neural Representation Framework for Tabular Data Imputation
• Causally-Informed Reinforcement Learning for Adaptive Emotion-Aware Social Media Recommendation
• FraudTransformer: Time-Aware GPT for Transaction Fraud Detection
• Prehospital triage of trauma patients: predicting major surgery using artificial intelligence as decision support
• Impute-MACFM: Imputation based on Mask-Aware Flow Matching
• SASD: Self-Attention for Small Datasets—A case study in smart villages
• Learn to Rank Risky Investors: A Case Study of Predicting Retail Traders' Behaviour and Profitability
• Improving Deep Tabular Learning
• Intrinsic Meets Extrinsic Fairness: Assessing the Downstream Impact of Bias Mitigation in Large Language Models
• Explainable AI for Infection Prevention and Control: Modeling CPE Acquisition and Patient Outcomes in an Irish Hospital with Transformers
• Large Language Models in Document Intelligence: A Comprehensive Survey, Recent Advances, Challenges, and Future Trends
• Towards Interpretable Deep Neural Networks for Tabular Data
• In-Context Learning Enhanced Credibility Transformer
• Probabilistic Pretraining for Neural Regression
• Prediction of mortality and resource utilization in critical care: a deep learning approach using multimodal electronic health records with natural language processing techniques
• A Systematic Review on the Generative AI Applications in Human Medical Genomics
• Robust Detection of Synthetic Tabular Data under Schema Variability
• Revisiting Deep Learning Models for Tabular Data
• Deep Learning for School Dropout Detection: A Comparison of Tabular and Graph-Based Models for Predicting At-Risk Students
• DeepGB-TB: A Risk-Balanced Cross-Attention Gradient-Boosted Convolutional Network for Rapid, Interpretable Tuberculosis Screening
• PyG 2.0: Scalable Learning on Real World Graphs
• FOUNDv2: Learning Unified User Quantized Tokenizers for User Representation
• ITUNLP at SemEval-2025 Task 8: Question-Answering over Tabular Data: A Zero-Shot Approach using LLM-Driven Code Generation
• PRAGMA: Revolut Foundation Model
• Self-supervised learning on gene expression data
• TabSyM: A Generative Pipeline for Small Multi-Cohort Omics Tabular Data
• Crafting Imperceptible On-Manifold Adversarial Attacks for Tabular Data
• MIRRAMS: Learning Robust Tabular Models under Unseen Missingness Shifts
• Universal Embeddings of Tabular Data
• Task-Agnostic Contrastive Pretraining for Relational Deep Learning
• REDELEX: A Framework for Relational Deep Learning Exploration
• Data Curation Matters: Model Collapse and Spurious Shift Performance Prediction from Training on Uncurated Text Embeddings
• Relational Deep Learning: Challenges, Foundations and Next-Generation Architectures
• Universal Laboratory Model: prognosis of abnormal clinical outcomes based on routine tests
• Learning based on neurovectors for tabular data: a new neural network approach
• Basis Transformers for Multi-Task Tabular Regression
• Benchmarking Multimodal AutoML for Tabular Data with Text Fields
• TabFlex: Scaling Tabular Learning to Millions with Linear Attention
• On the Robustness of Tabular Foundation Models: Test-Time Attacks and In-Context Defenses
• AI-Driven Vehicle Condition Monitoring with Cell-Aware Edge Service Migration
• Random at First, Fast at Last: NTK-Guided Fourier Pre-Processing for Tabular DL
• CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
• ACCESS DENIED INC: The First Benchmark Environment for Sensitivity Awareness
• NeuronSeek: On Stability and Expressivity of Task-driven Neurons
• TIME: TabPFN-Integrated Multimodal Engine for Robust Tabular-Image Learning
• X2Graph for Cancer Subtyping Prediction on Biological Tabular Data
• Does your graph need a confidence boost? Convergent boosted smoothing on graphs with tabular node features
• TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fields
• LLM Meeting Decision Trees on Tabular Data
• Bridging Electronic Health Records and Clinical Texts: Contrastive Learning for Enhanced Clinical Tasks
• SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training
• A Robust PPO-optimized Tabular Transformer Framework for Intrusion Detection in Industrial IoT Systems
• Dual-Agent Reinforcement Learning for Automated Feature Generation
• Make Still Further Progress: Chain of Thoughts for Tabular Data Leaderboard
• Harnessing LLMs Explanations to Boost Surrogate Models in Tabular Data Classification
• Latte: Transfering LLMs` Latent-level Knowledge for Few-shot Tabular Learning
• Griffin: Towards a Graph-Centric Relational Database Foundation Model
• LakeMLB: Data Lake Machine Learning Benchmark
• LEAP: A Rapid Neural Surrogate of Multi-Fluid MHD at Europa
• Attention-enabled Explainable AI for Bladder Cancer Recurrence Prediction
• ReSS: Learning Reasoning Models for Tabular Data Prediction via Symbolic Scaffold
• ALF: Advertiser Large Foundation Model for Multi-Modal Advertiser Understanding
• Enhancing Strawberry Yield Forecasting with Backcasted IoT Sensor Data and Machine Learning
• iStructTab: Structured Feature Sequencing for Multimodal Learning of Image and Tabular Data
• Adaptive Intrusion Detection System using Transformer-Based Neural Networks and Continual Learning Approach with Adversarial Investigation
• Against Opacity: Explainable AI and Large Language Models for Effective Digital Advertising
• Imputation-free Learning of Tabular Data with Missing Values using Incremental Feature Partitions in Transformer
• Deep Learning with Pretrained 'Internal World' Layers: A Gemma 3-Based Modular Architecture for Wildfire Prediction
• Deep Neural Networks and Tabular Data: A Survey
• Representation Learning for Tabular Data: A Comprehensive Survey
• Beyond Black-Box Predictions: Identifying Marginal Feature Effects in Tabular Transformer Networks
• Datum-wise Transformer for Synthetic Tabular Data Detection in the Wild
• Boosting Relational Deep Learning with Pretrained Tabular Models
• Deep Transformers for Analyzing BOF Steelmaking Data

Related

• Deep Learning with Tabular Data: A Self-supervised Approach
• Deep Learning within Tabular Data: Foundations, Challenges, Advances and Future Directions
• TabTranSELU: A transformer adaptation for solving tabular data
• TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fields
• PTab: Using the Pre-trained Language Model for Modeling Tabular Data