Why LLMs Are Bad at Synthetic Table Generation (and what to do about it)

2024/06/20 by Shengzhe Xu, Cho-Ting Lee, Xu, Shengzhe +9 · 2 citations
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Mathematics, Computing, and Information Processing #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2406.14541

openalex publication_date 2024/06/20 · openalex created_date 2024/06/22 · openalex updated_date 2026/07/28

Abstract

Synthetic data generation is integral to ML pipelines, e.g., to augment training data, replace sensitive information, and even to power advanced platforms like DeepSeek. While LLMs fine-tuned for synthetic data generation are gaining traction, synthetic table generation -- a critical data type in business and science -- remains under-explored compared to text and image synthesis. This paper shows that LLMs, whether used as-is or after traditional fine-tuning, are inadequate for generating synthetic tables. Their autoregressive nature, combined with random order permutation during fine-tuning, hampers the modeling of functional dependencies and prevents capturing conditional mixtures of distributions essential for real-world constraints. We demonstrate that making LLMs permutation-aware can mitigate these issues.

Cited by

• Doubling Your Data in Minutes: Ultra-fast Tabular Data Generation via LLM-Induced Dependency Graphs
• Not All Features Deserve Attention: Graph-Guided Dependency Learning for Tabular Data Generation with Language Models

Related

• Enhancing Table Representations with LLM-powered Synthetic Data Generation
• On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey
• FASTGEN: Fast and Cost-Effective Synthetic Tabular Data Generation with LLMs
• TableDreamer: Progressive and Weakness-guided Data Synthesis from Scratch for Table Instruction Tuning
• Synthetic Eggs in Many Baskets: The Impact of Synthetic Data Diversity on LLM Fine-Tuning