Evaluating Large Language Models with NeuBAROCO: Syllogistic Reasoning Ability and Human-like Biases

2023/06/21 by Risako Ando, Takanobu Morishita, Ando, Risako +7 · 4 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2306.12567

openalex publication_date 2023/06/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper investigates whether current large language models exhibit biases in logical reasoning, similar to humans. Specifically, we focus on syllogistic reasoning, a well-studied form of inference in the cognitive science of human deduction. To facilitate our analysis, we introduce a dataset called NeuBAROCO, originally designed for psychological experiments that assess human logical abilities in syllogistic reasoning. The dataset consists of syllogistic inferences in both English and Japanese. We examine three types of biases observed in human syllogistic reasoning: belief biases, conversion errors, and atmosphere effects. Our findings demonstrate that current large language models struggle more with problems involving these three types of biases.

Cited by

• Normative Reasoning in Large Language Models: A Comparative Benchmark from Logical and Modal Perspectives
• Assessing the Relational Abilities of Large Language Models and Large Reasoning Models
• From Indirect Object Identification to Syllogisms: Exploring Binary Mechanisms in Transformer Circuits
• BIS Reasoning 1.0: The First Large-Scale Japanese Benchmark for Belief-Inconsistent Syllogistic Reasoning

Related

• Exploring Reasoning Biases in Large Language Models Through Syllogism: Insights from the NeuBAROCO Dataset
• A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models
• A Systematic Analysis of Large Language Models as Soft Reasoners: The Case of Syllogistic Inferences
• Language models show human-like content effects on reasoning tasks
• BIS Reasoning 1.0: The First Large-Scale Japanese Benchmark for Belief-Inconsistent Syllogistic Reasoning