Benchmarking Abstract and Reasoning Abilities Through A Theoretical Perspective

2025/05/28 by Qingchuan Ma, Ma, Qingchuan, Yuhang Wu +5 · 1 citation
Computer Science · Social Sciences · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Intelligent Tutoring Systems and Adaptive Learning #Mathematics Education and Teaching Techniques #Teaching and Learning Programming

paper · pdf · doi:10.48550/arxiv.2505.23833

openalex publication_date 2025/05/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we aim to establish a simple, effective, and theoretically grounded benchmark for rigorously probing abstract reasoning in Large Language Models (LLMs). To achieve this, we first develop a mathematic framework that defines abstract reasoning as the ability to: (i) extract essential patterns independent of surface representations, and (ii) apply consistent rules to these abstract patterns. Based on this framework, we introduce two novel complementary metrics: \(\scoreGamma\) measures basic reasoning accuracy, while \(\scoreDelta\) quantifies a model's reliance on specific symbols rather than underlying patterns - a key indicator of true abstraction versus mere memorization. To implement this measurement, we design a benchmark: systematic symbol remapping in rule-based tasks, which forces models to demonstrate genuine pattern recognition beyond superficial token matching. Extensive LLM evaluations using this benchmark (commercial API models, 7B-70B, multi-agent) reveal:1) critical limitations in non-decimal arithmetic and symbolic reasoning; 2) persistent abstraction gaps despite chain-of-thought prompting; and 3) \(\scoreDelta\)'s effectiveness in robustly measuring memory dependence by quantifying performance degradation under symbol remapping, particularly highlighting operand-specific memorization. These findings underscore that current LLMs, despite domain-specific strengths, still lack robust abstract reasoning, highlighting key areas for future improvement.

Citations

• Qwen2.5 Technical Report
• Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions
• Combining Induction and Transduction for Abstract Reasoning
• GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
• OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data
• Logic-of-Thought: Injecting Logic into Contexts for Full Reasoning in Large Language Models
• The Llama 3 Herd of Models
• Gemma 2: Improving Open Language Models at a Practical Size
• DiscoveryBench: Towards Data-Driven Discovery with Large Language Models
• ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools
• A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners
• InternLM2 Technical Report
• Meaningful Learning: Enhancing Abstract Reasoning in Large Language Models via Generic Fact Guidance
• Yi: Open Foundation Models by 01.AI
• Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMs
• Neural networks for abstraction and reasoning: Towards broad generalization in machines
• Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks
• When can transformers reason with abstract symbols?
• Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement
• MuggleMath: Assessing the Impact of Query and Response Augmentation on Math Reasoning
• Scaling Relationship on Learning Mathematical Reasoning with Large Language Models
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Large Language Models as General Pattern Machines
• Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks
• Improving Factuality and Reasoning in Language Models through Multiagent Debate
• PaLM 2 Technical Report
• GPT-4 Technical Report
• LLaMA: Open and Efficient Foundation Language Models
• Towards Reasoning in Large Language Models: A Survey
• Emergent Analogical Reasoning in Large Language Models
• Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
• Distributing Accountability, Not Capability: Phase Separation and the LLM Workflow Quadrant in Autonomous AI Agent Architectures
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Training Verifiers to Solve Math Word Problems
• Measuring Mathematical Problem Solving With the MATH Dataset
• Neurosymbolic AI: The 3rd Wave
• Measuring abstract reasoning in neural networks
• Neural-Symbolic Learning and Reasoning: A Survey and Interpretation

Cited by

• Prior Knowledge Makes It Possible: From Sublinear Graph Algorithms to LLM Test-Time Methods

Related

• Large Language Models Are Not Strong Abstract Reasoners
• Meaningful Learning: Enhancing Abstract Reasoning in Large Language Models via Generic Fact Guidance
• RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
• Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
• Benchmarking Reasoning Robustness in Large Language Models