Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models

2024/11/19 by Laura Ruis, Ruis, Laura, Maximilian Mozes +18 · 24 voices · 10 citations
Computer Science · #Natural Language Processing Techniques #Topic Modeling #Semantic Web and Ontologies

paper · pdf · doi:10.48550/arxiv.2411.12580

Abstract

The capabilities and limitations of Large Language Models have been sketched out in great detail in recent years, providing an intriguing yet conflicting picture. On the one hand, LLMs demonstrate a general ability to solve problems. On the other hand, they show surprising reasoning gaps when compared to humans, casting doubt on the robustness of their generalisation strategies. The sheer volume of data used in the design of LLMs has precluded us from applying the method traditionally used to measure generalisation: train-test set separation. To overcome this, we study what kind of generalisation strategies LLMs employ when performing reasoning tasks by investigating the pretraining data they rely on. For two models of different sizes (7B and 35B) and 2.5B of their pretraining tokens, we identify what documents influence the model outputs for three simple mathematical reasoning tasks and contrast this to the data that are influential for answering factual questions. We find that, while the models rely on mostly distinct sets of data for each factual question, a document often has a similar influence across different reasoning questions within the same task, indicating the presence of procedural knowledge. We further find that the answers to factual questions often show up in the most influential data. However, for reasoning questions the answers usually do not show up as highly influential, nor do the answers to the intermediate reasoning steps. When we characterise the top ranked documents for the reasoning questions qualitatively, we confirm that the influential documents often contain procedural knowledge, like demonstrating how to obtain a solution using formulae or code. Our findings indicate that the approach to reasoning the models use is unlike retrieval, and more like a generalisable strategy that synthesises procedural knowledge from documents doing a similar form of reasoning.

Cited by

• AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong Pretraining Data Selection
• AIR: Post-training Data Selection for Reasoning via Attention Head Influence
• Training Language Models via Neural Cellular Automata
• Refine Thought: A Test-Time Inference Method for Embedding Model Reasoning
• Influence Functions for Efficient Data Selection in Reasoning
• Train Once, Answer All: Many Pretraining Experiments for the Cost of One
• Learning When to Plan: Efficiently Allocating Test-Time Compute for LLM Agents
• Beyond Memorization: Reasoning-Driven Synthesis as a Mitigation Strategy Against Benchmark Contamination
• Concept-TRAK: Understanding how diffusion models learn concepts through concept-level attribution
• BOW: Reinforcement Learning for Bottlenecked Next Word Prediction

Discussions

• Procedural knowledge in pretraining drives reasoning in large language models [hn, 248 points, 101 comments]
• 🚨 LLMs can learn to reason from procedural knowledge in pretraining data! 🚨 I particularly enjoy research where the evidence contradicts our initial hypothesis. If you're interested in LLM reasoning [bsky, 67 points, 4 comments]
• If you made it this far, thanks for reading, and check out the paper or blogpost! Paper: arxiv.org/abs/2411.12580 Blogpost: lauraruis.github.io/2024/11/10/i... Demo (more data to come!): lauraruis.gi [bsky, 53 points, 6 comments]
• 🚨 Alert: Very Readable Paper 🚨 The “do LLMs think?” question always bugged me because I have no idea what that means. This paper focuses narrowly on, “do LLMs learn **how to do** things”. Unlike m [bsky, 46 points, 3 comments]
• Some very nice work from Cohere and UCL using influence functions to analyze math reasoning abilities in LLMs. Factual queries turn up docs containing the facts, but reasoning queries turn up similar [bsky, 16 points, 0 comments]
• Another cracking paper by @lauraruis.bsky.social (and colleagues!) which has deeply affected how I think about LLM capabilities. So happy to have been able to collaborate with her on this. Give it a r [bsky, 9 points, 1 comments]
• arxiv.org/abs/2411.12580 [bsky, 8 points, 0 comments]
• incredibly cool work on demonstrating models truly do reason by Laura Ruis arxiv.org/abs/2411.12580 [bsky, 7 points, 1 comments]
• New #AI research suggests #LLM pretraining encourages the development of "reasoning" skills by learning general procedures, rather than just memorizing answers. [bsky, 6 points, 2 comments]
• Didn't check the methodology well enough, but this is an interesting paper (via HN) saying LLMs learn reasoning heuristics, not answers. For math they draw on “procedural” data—methods and formulas—r [bsky, 4 points, 0 comments]
• Finally! A practical method for doing influence analysis on LLMs. Shows how elements of the *training data* influence LLM output. And a cool application showing how LLMs do math. Important work. arx [bsky, 3 points, 0 comments]
• Someone just tried to gotcha @abeba.bsky.social 's assertion of little to no evidence of reasoning in LLMs by linking to this, which I think is literally just trying to position 'prompt-reflective-ret [bsky, 2 points, 1 comments]
• Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models [hn, 2 points, 0 comments]
• New paper suggests that procedural knowledge in pretraining data influences reasoning capabilities. Unlike factual Q&A, reasoning tasks rely on documents showing how to solve problems (e.g., formulas [bsky, 1 points, 0 comments]
• Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models (arxiv.org) Main Link | Discussion [bsky, 1 points, 0 comments]
• Procedural knowledge in pretraining drives reasoning in large language models view on hacker news [bsky, 0 points, 0 comments]
• That's not what the actual research says arxiv.org/pdf/2411.12580 [bsky, 0 points, 1 comments]
• Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models https://arxiv.org/abs/2411.12580 (https://news.ycombinator.com/item?id=42289310) [bsky, 0 points, 0 comments]
• Procedural knowledge in pretraining drives reasoning in large language models https://arxiv.org/abs/2411.12580 [comments] [191 points] [bsky, 0 points, 0 comments]
• arxiv.org/abs/2411.12580 Very interesting paper. I think far more work like this is necessary to convince me that these coced up stochastic parrots can indeed learn new knowledge or engage in trustwo [bsky, 0 points, 0 comments]
• arxiv.org/abs/2411.12580 Another great cargo cult “paper.” Burning gpu cycles to move lights on the runway. [bsky, 0 points, 0 comments]
• Procedural knowledge in pretraining drives reasoning in large language models https://arxiv.org/abs/2411.12580 https://news.ycombinator.com/item?id=42289310 [bsky, 0 points, 0 comments]
• Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models [bsky, 0 points, 0 comments]
• Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models https://arxiv.org/abs/2411.12580 (https://news.ycombinator.com/item?id=42289310) [bsky, 0 points, 0 comments]

Related

• Hierarchical Deconstruction of LLM Reasoning: A Graph-Based Framework for Analyzing Knowledge Utilization
• What Do Learning Dynamics Reveal About Generalization in LLM Reasoning?
• Specializing Smaller Language Models towards Multi-Step Reasoning
• LIMO: Less is More for Reasoning
• Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey