CausalARC: Abstract Reasoning with Causal World Models

2025/09/03 by Maasch, Jacqueline, Kalantari, John, Khezeli, Kia
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2509.03636

Abstract

On-the-fly reasoning often requires adaptation to novel problems under limited data and distribution shift. This work introduces CausalARC: an experimental testbed for AI reasoning in low-data and out-of-distribution regimes, modeled after the Abstraction and Reasoning Corpus (ARC). Each CausalARC reasoning task is sampled from a fully specified causal world model, formally expressed as a structural causal model. Principled data augmentations provide observational, interventional, and counterfactual feedback about the world model in the form of few-shot, in-context learning demonstrations. As a proof-of-concept, we illustrate the use of CausalARC for four language model evaluation settings: (1) abstract reasoning with test-time training, (2) counterfactual reasoning with in-context learning, (3) program synthesis, and (4) causal discovery with logical reasoning. Within- and between-model performance varied heavily across tasks, indicating room for significant improvement in language model reasoning.

Citations

• RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
• The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
• Compositional Causal Reasoning Evaluation in Language Models
• PhantomWiki: On-Demand Datasets for Reasoning and Retrieval Evaluation
• Since Faithfulness Fails: The Performance Limits of Neural Causal Discovery
• ARC Prize 2024: Technical Report
• The Surprising Effectiveness of Test-Time Training for Few-Shot Learning
• Combining Induction and Transduction for Abstract Reasoning
• GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
• Reasoning Elicitation in Language Models via Counterfactual Feedback
• H-ARC: A Robust Estimate of Human Performance on the Abstraction and Reasoning Corpus Benchmark
• Does Reasoning Emerge? Examining the Probabilities of Causation in Large Language Models
• Addressing the Abstraction and Reasoning Corpus via Procedural Example Generation
• Robust agents learn causal world models
• CLadder: Assessing Causal Reasoning in Language Models
• Can Large Language Models Infer Causation from Correlation?
• Reasoning with Language Model is Planning with World Model
• Is ChatGPT a Good Causal Reasoner? A Comprehensive Evaluation
• The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domain
• A Comprehensive Survey on Test-Time Adaptation Under Distribution Shifts
• Larger language models do in-context learning differently
• Causal-Discovery Performance of ChatGPT in the context of Neuropathic Pain Diagnosis
• Efficient Test-Time Model Adaptation without Forgetting
• Language Models are Few-Shot Learners
• Review of Causal Discovery Methods Based on Graphical Models
• Human few-shot learning of compositional instructions
• World Models
• Probabilities of Causation: Bounds and Identification
• General agents contain world models

Related

• Causal Reasoning in Pieces: Modular In-Context Learning for Causal Discovery
• Is Knowledge All Large Language Models Needed for Causal Reasoning?
• CausalEval: Towards Better Causal Reasoning in Language Models
• Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?
• Causal Evaluation of Language Models