CLEAR-3K: Assessing Causal Explanatory Capabilities in Language Models

2025/06/20 by N.-S. Liu, Richard G. Baraniuk, Liu, Naiming +3
Computer Science · #Computation and Language (cs.CL) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2506.17180

openalex publication_date 2025/06/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce CLEAR-3K, a dataset of 3,000 assertion-reasoning questions designed to evaluate whether language models can determine if one statement causally explains another. Each question present an assertion-reason pair and challenge language models to distinguish between semantic relatedness and genuine causal explanatory relationships. Through comprehensive evaluation of 21 state-of-the-art language models (ranging from 0.5B to 72B parameters), we identify two fundamental findings. First, language models frequently confuse semantic similarity with causality, relying on lexical and semantic overlap instead of inferring actual causal explanatory relationships. Second, as parameter size increases, models tend to shift from being overly skeptical about causal relationships to being excessively permissive in accepting them. Despite this shift, performance measured by the Matthews Correlation Coefficient plateaus at just 0.55, even for the best-performing models.Hence, CLEAR-3K provides a crucial benchmark for developing and evaluating genuine causal reasoning in language models, which is an essential capability for applications that require accurate assessment of causal relationships.

Citations

• Qwen3 Technical Report
• Qwen2.5 Technical Report
• Phi-4 Technical Report
• The Llama 3 Herd of Models
• MalAlgoQA: Pedagogical Evaluation of Counterfactual Reasoning in Large Language Models and Implications for AI in Education
• Automated Long Answer Grading with RiceChem Dataset
• CASA: Causality-driven Argument Sufficiency Assessment
• Can Large Language Models Infer Causation from Correlation?
• CLASS: A Design Framework for building Intelligent Tutoring Systems based on Learning Science principles
• Fact or Fiction: Verifying Scientific Claims
• Know What You Don't Know: Unanswerable Questions for SQuAD
• FEVER: a large-scale dataset for Fact Extraction and VERification
• A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• A large annotated corpus for learning natural language inference
• CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge

Related

• ExpliCa: Evaluating Explicit Causal Reasoning in Large Language Models
• A Critical Review of Causal Reasoning Benchmarks for Large Language Models
• Causal Distillation: Transferring Structured Explanations from Large to Compact Language Models
• CausalEval: Towards Better Causal Reasoning in Language Models
• Causal Evaluation of Language Models