TabGemma: Text-Based Tabular ICL via LLM using Continued Pretraining and Retrieval

2025/11/05 by Günther Schindler, Maximilian Schambach, Schindler, Günther +5 · 2 citations
Computer Science · Social Sciences · #Computational and Text Analysis Methods #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning in Healthcare #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2511.03570

openalex publication_date 2025/11/05 · openalex created_date 2025/11/07 · openalex updated_date 2026/07/28

Abstract

We study LLMs for tabular prediction with mixed text, numeric, and categorical fields. We introduce TabGemma, a schema-agnostic in-context learner that treats rows as sequences and tackles two practical hurdles when adapting pretrained LLMs for tabular predictions: unstable numeric tokenization and limited context size. We propose to canonicalize numbers via signed scientific notation and continue pretraining of a 12B Gemma 3 model with a target imputation objective using a large-scale real world dataset. For inference, we use a compact n-gram-based retrieval to select informative exemplars that fit within a 128k-token window. On semantically rich benchmarks, TabGemma establishes a new state of the art on classification across low- and high-data regimes and improves monotonically with more context rows. For regression, it is competitive at small sample sizes but trails conventional approaches as data grows. Our results show that LLMs can be effective tabular in-context learners on highly semantic tasks when paired with dedicated numeric handling and context retrieval, while motivating further advances in numeric modeling and long-context scaling.

Citations

• TabArena: A Living Benchmark for Machine Learning on Tabular Data
• ConTextTab: A Semantics-Aware Tabular In-Context Learner
• Gemma 3 Technical Report
• TabICL: A Tabular Foundation Model for In-Context Learning on Large Data
• TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling
• Better by Default: Strong Pre-Tuned MLPs and Boosted Trees on Tabular Data
• Revisiting Nearest Neighbor for Tabular Data: A Deep Tabular Baseline Two Decades Later
• Large Scale Transfer Learning for Tabular Data via Language Modeling
• CARTE: Pretraining and Transfer for Tabular Learning
• The Faiss library
• Vectorizing string entries for data processing on tables: when are larger language models better?
• From Supervised to Generative: A Novel Paradigm for Tabular Deep Learning with Large Language Models
• TabLib: A Dataset of 627M Tables with Context
• TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023
• XTab: Cross-table Pretraining for Tabular Transformers
• How well do Large Language Models perform in Arithmetic tasks?
• TabLLM: Few-shot Classification of Tabular Data with Large Language Models
• TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second
• LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks
• Revisiting Deep Learning Models for Tabular Data
• AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data
• Towards Benchmarking Foundation Models for Tabular Data With Text

Cited by

• MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Image
• Enhancing Tabular Learners with Context-Aware Semantic Embeddings

Related

• TabDPT: Scaling Tabular Foundation Models on Real Data
• Scalable In-Context Learning on Tabular Data via Retrieval-Augmented Large Language Models
• ConTextTab: A Semantics-Aware Tabular In-Context Learner
• TabGen-ICL: Residual-Aware In-Context Example Selection for Tabular Data Generation
• Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning