2024/06/06 by Chaoxu Pang, Pang, Chaoxu, Yixuan Cao +5 · 2 citations
Computer Science · Decision Sciences · #Advanced Text Analysis Techniques #Computation and Language (cs.CL) #Data Quality and Management #FOS: Computer and information sciences #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2406.04113
openalex publication_date 2024/06/06 · openalex created_date 2024/06/08 · openalex updated_date 2026/07/28
Tables are recognized for their high information density and widespread usage, serving as essential sources of information. Seeking information from tables (TIS) is a crucial capability for Large Language Models (LLMs), serving as the foundation of knowledge-based Q&A systems. However, this field presently suffers from an absence of thorough and reliable evaluation. This paper introduces a more reliable benchmark for Table Information Seeking (TabIS). To avoid the unreliable evaluation caused by text similarity-based metrics, TabIS adopts a single-choice question format (with two options per question) instead of a text generation format. We establish an effective pipeline for generating options, ensuring their difficulty and quality. Experiments conducted on 12 LLMs reveal that while the performance of GPT-4-turbo is marginally satisfactory, both other proprietary and open-source models perform inadequately. Further analysis shows that LLMs exhibit a poor understanding of table structures, and struggle to balance between TIS performance and robustness against pseudo-relevant tables (common in retrieval-augmented systems). These findings uncover the limitations and potential challenges of LLMs in seeking information from tables. We release our data and code to facilitate further research in this field.