The Geometry of Numerical Reasoning: Language Models Compare Numeric Properties in Linear Subspaces

2024/10/17 by Ahmed Oumar El-Shangiti, Takaharu Hiraoka, El-Shangiti, Ahmed Oumar +7 · 3 citations
Computer Science · #Computation and Language (cs.CL) #Constraint Satisfaction and Optimization #FOS: Computer and information sciences #Model-Driven Software Engineering Techniques #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2410.13194

openalex publication_date 2024/10/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper investigates whether large language models (LLMs) utilize numerical attributes encoded in a low-dimensional subspace of the embedding space when answering questions involving numeric comparisons, e.g., Was Cristiano born before Messi? We first identified, using partial least squares regression, these subspaces, which effectively encode the numerical attributes associated with the entities in comparison prompts. Further, we demonstrate causality, by intervening in these subspaces to manipulate hidden states, thereby altering the LLM's comparison outcomes. Experiments conducted on three different LLMs showed that our results hold across different numerical attributes, indicating that LLMs utilize the linearly encoded information for numerical reasoning.

Cited by

• Interpreting Multi-Attribute Confounding through Numerical Attributes in Large Language Models
• Shape Happens: Automatic Feature Manifold Discovery in LLMs via Supervised Multi-Dimensional Scaling
• Can Language Models Handle a Non-Gregorian Calendar? The Case of the Japanese wareki

Related

• Language Models Encode Numbers Using Digit Representations in Base 10
• Exploring Internal Numeracy in Language Models: A Case Study on ALBERT
• Number Representations in LLMs: A Computational Parallel to Human Perception
• Exploring the Numerical Reasoning Capabilities of Language Models: A Comprehensive Analysis on Tabular Data
• Interpreting Multi-Attribute Confounding through Numerical Attributes in Large Language Models