ExpliCa: Evaluating Explicit Causal Reasoning in Large Language Models

2025/02/21 by Martina Miliani, Miliani, Martina, Serena Auriemma +11 · 1 citation
Computer Science · #68T07 #68T50 #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.7 #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2502.15487

openalex publication_date 2025/02/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) are increasingly used in tasks requiring interpretive and inferential accuracy. In this paper, we introduce ExpliCa, a new dataset for evaluating LLMs in explicit causal reasoning. ExpliCa uniquely integrates both causal and temporal relations presented in different linguistic orders and explicitly expressed by linguistic connectives. The dataset is enriched with crowdsourced human acceptability ratings. We tested LLMs on ExpliCa through prompting and perplexity-based metrics. We assessed seven commercial and open-source LLMs, revealing that even top models struggle to reach 0.80 accuracy. Interestingly, models tend to confound temporal relations with causal ones, and their performance is also strongly influenced by the linguistic order of the events. Finally, perplexity-based scores and prompting performance are differently affected by model size.

Cited by

• Do Large Language Models Show Biases in Causal Learning? Insights from Contingency Judgment

Related

• A Critical Review of Causal Reasoning Benchmarks for Large Language Models
• Causal Reasoning and Large Language Models: Opening a New Frontier for Causality
• A Survey on Enhancing Causal Reasoning Ability of Large Language Models
• Understanding Causality with Large Language Models: Feasibility and Opportunities
• Can Large Language Models Infer Causal Relationships from Real-World Text?