Talking Trees: Reasoning-Assisted Induction of Decision Trees for Tabular Data

2025/09/25 by George Yakushev, Yakushev, George, Alina Shutova +8
Computer Science · #Advanced Graph Neural Networks #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.2509.21465

openalex publication_date 2025/09/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Tabular foundation models are becoming increasingly popular for low-resource tabular problems. These models make up for small training datasets by pretraining on large volumes of synthetic data. The prior knowledge obtained via pretraining provides the exceptional performance, but the resulting model becomes a black box that is difficult to interpret and costly for inference. In this work, we explore an alternative strategy: using reasoning-capable LLMs to induce decision trees for small tabular datasets in an agentic setup. We design a minimal set of tools for constructing, analyzing, and manipulating decision trees. Equipped with these tools, the LLM combines its prior knowledge with learning from data to produce a lightweight decision tree that outperforms CART and recent non-greedy tree learners and remains competitive with tree ensembles on low-resource tabular problems. While a single agentic decision tree is competitive with state-of-the-art black box models, it also comes with a human-readable reasoning trace that can be checked for biases and data leaks. Furthermore, the reasoning-based LLM's creation process allows for additional human input to be incorporated into the tree without it being captured in data.

Citations

• MachineLearningLM: Scaling Many-shot In-context Learning via Continued Pretraining
• LimiX: Unleashing Structured-Data Modeling Capability for Generalist Intelligence
• GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models
• Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety
• TabArena: A Living Benchmark for Machine Learning on Tabular Data
• On Finetuning Tabular Foundation Models
• CoT Red-Handed: Stress Testing Chain-of-Thought Monitoring
• LLM Meeting Decision Trees on Tabular Data
• MLZero: A Multi-Agent System for End-to-end Machine Learning Automation
• Make Still Further Progress: Chain of Thoughts for Tabular Data Leaderboard
• LLM-FE: Automated Feature Engineering for Tabular Data with LLMs as Evolutionary Optimizers
• Exploring LLM Agents for Cleaning Tabular Machine Learning Datasets
• AIDE: AI-Driven Exploration in the Space of Code
• TabICL: A Tabular Foundation Model for In-Context Learning on Large Data
• TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems
• Zero-Shot Decision Tree Construction via Large Language Models
• AutoDCWorkflow: LLM-based Data Cleaning Workflow Auto-Generation and Benchmark
• TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling
• TabDPT: Scaling Tabular Foundation Models on Real Data
• Data Cleaning Using Large Language Models
• ABCFair: an Adaptable Benchmark approach for Comparing Fairness Methods
• The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies
• Large Scale Transfer Learning for Tabular Data via Language Modeling
• Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning
• AirGapAgent: Protecting Privacy-Conscious Conversational Agents
• Identifying the Risks of LM Agents with an LM-Emulated Sandbox
• Breiman meets Bellman: Non-Greedy Decision Trees with MDPs
• Counterfactually Auditable Lifecycle Certification for Autonomous Agents
• Language models are weak learners
• Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
• Toolformer: Language Models Can Teach Themselves to Use Tools
• TabLLM: Few-shot Classification of Tabular Data with Large Language Models
• Automatic Chain of Thought Prompting in Large Language Models
• TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second
• Large Language Models are Zero-Shot Reasoners
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Retiring Adult: New Datasets for Fair Machine Learning
• Learning Transferable Visual Models From Natural Language Supervision
• Array programming with NumPy
• Language Models are Few-Shot Learners
• SciPy 1.0: fundamental algorithms for scientific computing in Python
• The Frontiers of Fairness in Machine Learning
• A Reductions Approach to Fair Classification
• A comparative study of fairness-enhancing interventions in machine learning
• Equality of Opportunity in Supervised Learning
• Peeking Inside the Black Box: Visualizing Statistical Learning with Plots of Individual Conditional Expectation
• Peeking Inside the Black Box: Visualizing Statistical Learning With Plots of Individual Conditional Expectation
• Fairness Through Awareness
• Greedy function approximation: A gradient boosting machine.
• A Survey of LLM × DATA
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• LLM Meeting Decision Trees on Tabular Data
• "Oh LLM, I'm Asking Thee, Please Give Me a Decision Tree": Zero-Shot Decision Tree Induction and Embedding with Large Language Models
• Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning
• Improving the Validity of Decision Trees as Explanations
• Utilizing Training Data to Improve LLM Reasoning for Tabular Understanding