Language Models Use Trigonometry to Do Addition

2025/02/02 by Subhash Kantamneni, Max Tegmark, Kantamneni, Subhash +1 · 33 voices · 17 citations
Computer Science · Arts and Humanities · #Natural Language Processing Techniques #Syntax, Semantics, Linguistic Variation #Speech and dialogue systems

paper · pdf · doi:10.48550/arxiv.2502.00873

Abstract

Mathematical reasoning is an increasingly important indicator of large language model (LLM) capabilities, yet we lack understanding of how LLMs process even simple mathematical tasks. To address this, we reverse engineer how three mid-sized LLMs compute addition. We first discover that numbers are represented in these LLMs as a generalized helix, which is strongly causally implicated for the tasks of addition and subtraction, and is also causally relevant for integer division, multiplication, and modular arithmetic. We then propose that LLMs compute addition by manipulating this generalized helix using the "Clock" algorithm: to solve a+b, the helices for a and b are manipulated to produce the a+b answer helix which is then read out to model logits. We model influential MLP outputs, attention head outputs, and even individual neuron preactivations with these helices and verify our understanding with causal interventions. By demonstrating that LLMs represent numbers on a helix and manipulate this helix to perform addition, we present the first representation-level explanation of an LLM's mathematical capability.

Cited by

• Are Arithmetic Heuristic Neurons Form-Invariant? A Mechanistic Analysis of Symbols, Text, and Code in LLMs
• Convergent Evolution: How Different Language Models Learn Similar Number Representations
• How Many Features Can a Language Model Store Under the Linear Representation Hypothesis?
• When Models Manipulate Manifolds: The Geometry of a Counting Task
• From Memorization to Reasoning in the Spectrum of Loss Curvature
• Why Can't Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls
• LLM Social Simulations Are a Promising Research Method
• In-Context Algebra
• Cognitive Maps in Language Models: A Mechanistic Analysis of Spatial Planning
• SOM Directions are Better than One: Multi-Directional Refusal Suppression in Language Models
• Interpreting Multi-Attribute Confounding through Numerical Attributes in Large Language Models
• LLMs Process Lists With General Filter Heads
• Unravelling the Mechanisms of Manipulating Numbers in Language Models
• The Geometry of Reasoning: Flowing Logics in Representation Space
• Language Models Do Not Embed Numbers Continuously
• MathBode: Measuring the Stability of LLM Reasoning using Frequency Response
• Modular Arithmetic: Language Models Solve Math Digit by Digit

Discussions

• this actually isn't true! arxiv.org/abs/2502.008... [bsky, 104 points, 6 comments]
• arxiv.org/abs/2502.00873 [bsky, 24 points, 0 comments]
• woah cool paper arxiv.org/abs/2502.00873 [bsky, 21 points, 1 comments]
• "Language Models Use Trigonometry to Do Addition" arxiv.org/pdf/2502.00873 This is pretty remarkable. Who could ever have predicted that LLMs (or anyone) would do it like that? [bsky, 13 points, 3 comments]
• We've found they use trigonometry for everything. Which as it turns out is *extremely* good for large numbers, and accurate (but yes, inefficient) for small numbers. But now they can just code even sm [bsky, 6 points, 1 comments]
• Like we *know* how LLMs do math (very weirdly), and it involves internal trigonometry arxiv.org/abs/2502.00873 [bsky, 5 points, 1 comments]
• arxiv.org/abs/2502.00873 [bsky, 5 points, 1 comments]
• arxiv.org/abs/2502.00873 [bsky, 5 points, 0 comments]
• sure, i believe it's this one by max tegmark et al (also a really good paper in terms of leading you to a ton of other interesting papers) [bsky, 4 points, 2 comments]
• New preprint suggests large language models use trigonometry to do addition: arxiv.org/pdf/2502.00873? [bsky, 4 points, 0 comments]
• They’re strange smarts. But they’re not the sort of Markovian system people seem to think they are. There’s more complex and sophisticated circuitry in there, and it is truly fascinating. Here are som [bsky, 4 points, 2 comments]
• I think this is an interesting example in that we do kind of have enough info pretty good gueses about how chagpt does addition. Namely something like this : arxiv.org/abs/2502.00873 But thats like on [bsky, 3 points, 1 comments]
• arxiv.org/abs/2502.00873 [bsky, 3 points, 1 comments]
• this one's actually not true, and it's quite interesting: LLMs perform addition by manipulating helices with trigonometry arxiv.org/abs/2502.00873 [bsky, 2 points, 1 comments]
• it's this one. effectively you're running a search on "every algorithm you might try to use with vectors to perform integer addition", and it turns out this is what that search finds arxiv.org/abs/250 [bsky, 2 points, 0 comments]
• Language Models Use Trigonometry to Do Addition [hn, 1 points, 3 comments]
• Maybe implicates the interaction of tokenization with the learned multiplication algorithm? This paper on numeric representation in LLMs floated by the other day, kinda interesting: arxiv.org/abs/2502 [bsky, 1 points, 1 comments]
• *I wonder what it physically looks like when PEOPLE add numbers from 1 to 99 in their heads "Mental arithmetic," that must be awful gooey and messy https://arxiv.org/abs/2502.00873 [bsky, 1 points, 0 comments]
• Following on arxiv.org/pdf/2502.00873, I think this speaks volumes. much better to learn the right abstractions. And where the right abstractions remain more contested, things will probably remain dic [bsky, 1 points, 3 comments]
• LLMs actually learns addition itself and uses a helix or dial and trigonometry to fashion a very rough numeric calculator. The paper is here arxiv.org/abs/2502.00873 [bsky, 1 points, 0 comments]
• LLMs are the product of a great crime, hallucinate more than would make them really useful for most tasks, but they do really do math. arxiv.org/abs/2502.00873 [bsky, 1 points, 1 comments]
• Direct link. No dog in this hunt, just passing by, :) arxiv.org/abs/2502.00873 [bsky, 1 points, 1 comments]
• Which you see in papers like arxiv.org/abs/2502.00873 [bsky, 1 points, 1 comments]
• …consider arxiv.org/abs/2502.00873, which is an investigation into internal representations of numbers in LLMs. There’s some evidence the models construct “helix” representations of the numbers in the [bsky, 1 points, 1 comments]
• yeah but really interesting internal models can emerge from that substrate! like for example this is a paper about LLMs containing internal geometric representations (as helices) of the number line, w [bsky, 0 points, 0 comments]
• This is interesting (and of course in need of expert peer view). via @dailynous.com arxiv.org/abs/2502.00873 [bsky, 0 points, 1 comments]
• "LLMs use trigonometry to perform addition". I view this as an astonishing and fascinating analysis. #AIResearch #LLMs #OtherMinds arxiv.org/abs/2502.00873 [bsky, 0 points, 0 comments]
• Language Models Do Addition Using Helices — wtf: "Mathematical reasoning is an increasingly important indicator of large language model (LLM) capabilities, yet we lack understanding of how LLMs proce [bsky, 0 points, 0 comments]
• arxiv.org/abs/2502.00873 [bsky, 0 points, 0 comments]
• This: arxiv.org/abs/2502.00873 is what I'm thinking of. Yes, it's a pre-print. But if correct, llms can evolve internal algorithms to perform addition. And even if wrong, can you truly claim to unders [bsky, 0 points, 0 comments]
• Someone has just indirectly put this arxiv.org/abs/2502.00873 in my timeline and I can't tell whether my howl is greater as a mathematician or as a lapsed sort-of-Popperian [bsky, 0 points, 1 comments]
• Pick up the pace Rocket Surgery! Brain Science is taking interesting steps: arxiv.org/abs/2502.00873 [bsky, 0 points, 0 comments]
• It is about this paper, for the curious -> arxiv.org/abs/2502.00873 [bsky, 0 points, 0 comments]

Related

• Interpreting and Improving Large Language Models in Arithmetic Calculation
• Modular Arithmetic: Language Models Solve Math Digit by Digit
• Language Models Encode the Value of Numbers Linearly
• Addition in Four Movements: Mapping Layer-wise Information Trajectories in LLMs
• Do Large Language Models Truly Grasp Addition? A Rule-Focused Diagnostic Using Two-Integer Arithmetic