Structural Deep Encoding for Table Question Answering

2025/03/03 by Raphaël Mouravieff, Mouravieff, Raphaël, Benjamin Piwowarski +3 · 2 citations
Computer Science · #Advanced Text Analysis Techniques #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2503.01457

openalex publication_date 2025/03/03 · openalex created_date 2025/10/12 · openalex updated_date 2026/07/28

Abstract

Although Transformers-based architectures excel at processing textual information, their naive adaptation for tabular data often involves flattening the table structure. This simplification can lead to the loss of essential inter-dependencies between rows, columns, and cells, while also posing scalability challenges for large tables. To address these issues, prior works have explored special tokens, structured embeddings, and sparse attention patterns. In this paper, we conduct a comprehensive analysis of tabular encoding techniques, which highlights the crucial role of attention sparsity in preserving structural information of tables. We also introduce a set of novel sparse attention mask designs for tabular data, that not only enhance computational efficiency but also preserve structural integrity, leading to better overall performance.

Cited by

• Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and Evaluation
• Granite Embedding R2 Models

Related

• TableFormer: Robust Transformer Modeling for Table-Text Encoding
• Bridging the Gap: Deciphering Tabular Data Using Large Language Model
• TabFlash: Efficient Table Understanding with Progressive Question Conditioning and Token Focusing
• Table Retrieval May Not Necessitate Table-specific Model Design
• Polynomial-based Self-Attention for Table Representation learning