Logic Haystacks: Probing LLMs Long-Context Logical Reasoning (Without Easily Identifiable Unrelated Padding)

2025/02/24 by Damien Sileo, Sileo, Damien · 2 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2502.17169

openalex publication_date 2025/02/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large language models demonstrate promising long context processing capabilities, with recent models touting context windows close to one million tokens. However, the evaluations supporting these claims often involve simple retrieval tasks or synthetic tasks padded with irrelevant text, which the models may easily detect and discard. In this work, we generate lengthy simplified English text with first-order logic representations spanning up to 2048 clauses (around 25k GPT-4 tokens). We formulate an evaluation task with evidence retrieval for contradiction detection. The long, homogeneous text is filled with distractors that are both hard to distinguish from relevant evidences and provably not interfering with them. Our evaluation of evidence retrieval shows that the effective context window is much smaller with realistic distractors, already crumbling at 128 clauses.

Cited by

• Reasoning Core: A Scalable RL Environment for LLM Symbolic Reasoning
• Reasoning Core: A Scalable Procedural Data Generation Suite for Symbolic Pre-training and Post-Training

Related

• BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
• Do Large Language Models Understand Logic or Just Mimick Context?
• NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densities
• Disentangling Logic: The Role of Context in Large Language Model Reasoning Capabilities
• Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacks