LIMO: Less is More for Reasoning

2025/02/05 by Y. M. Ye, Yixin Ye, Ye, Yixin +11 · 28 voices · 132 citations
Computer Science · Decision Sciences · #Modeling and Simulation Systems #Simulation Techniques and Applications

paper · pdf · doi:10.48550/arxiv.2502.03387

Abstract

We challenge the prevailing assumption that complex reasoning in large language models (LLMs) necessitates massive training data. We demonstrate that sophisticated mathematical reasoning can emerge with only a few examples. Specifically, through simple supervised fine-tuning, our model, LIMO, achieves 63.3% accuracy on AIME24 and 95.6% on MATH500, surpassing previous fine-tuned models (6.5% on AIME24, 59.2% on MATH500) while using only 1% of the training data required by prior approaches. Furthermore, LIMO exhibits strong out-of-distribution generalization, achieving a 45.8% absolute improvement across diverse benchmarks, outperforming models trained on 100x more data. Synthesizing these findings, we propose the Less-Is-More Reasoning Hypothesis (LIMO Hypothesis): In foundation models where domain knowledge has been comprehensively encoded during pre-training, sophisticated reasoning can emerge through minimal but strategically designed demonstrations of cognitive processes. This hypothesis suggests that the threshold for eliciting complex reasoning is not dictated by task complexity but rather by two key factors: (1) the completeness of the model's pre-trained knowledge base and (2) the effectiveness of post-training examples in serving as "cognitive templates" that guide reasoning.

Citations

• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thought
• GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
• AlphaMath Almost Zero: Process Supervision without Process
• Tree of Thoughts: Deliberate Problem Solving with Large Language Models

Cited by

• Learning to Reason for Factuality
• Artificial Intelligence for Mathematical Reasoning: An Integrated Survey of Language Models, Neuro-symbolic Systems, and Verified Discovery
• DataFlow-Harness: A Grounded Code-Agent Platform for Constructing Editable LLM Data Pipelines
• Implicit Actor Critic Coupling via a Supervised Learning Framework for RLVR
• Learning to Reason in 13 Parameters
• The Molecular Structure of Thought: Mapping the Topology of Long Chain-of-Thought Reasoning
• K2-Think: A Parameter-Efficient Reasoning System
• Reinforcement Learning Teachers of Test Time Scaling
• Reasoning Models Can Be Effective Without Thinking
• Wider or Deeper? Scaling LLM Inference-Time Compute with Adaptive Branching Tree Search
• s1: Simple test-time scaling
• CME-CAD: Heterogeneous Collaborative Multi-Expert Reinforcement Learning for CAD Code Generation
• VideoZoomer: Reinforcement-Learned Temporal Focusing for Long Video Reasoning
• Shape of Thought: When Distribution Matters More than Correctness in Reasoning Tasks
• PathScale-R1: Cross-scale Reasoning for Pathological Image Analysis
• Less Data, Better Alignment: Data-Centric Multi-Evaluator Agreement for Preference Optimization
• Where Did This Sentence Come From? Tracing Provenance in LLM Reasoning Distillation
• OpenDataArena: A Fair and Open Arena for Benchmarking Post-Training Dataset Value
• ADHint: Adaptive Hints with Difficulty Priors for Reinforcement Learning
• AIR: Post-training Data Selection for Reasoning via Attention Head Influence
• TraPO: A Semi-Supervised Reinforcement Learning Framework for Boosting LLM Reasoning
• VulnLLM-R: Specialized Reasoning LLM with Agent Scaffold for Vulnerability Detection
• Knowledge Graph Augmented Large Language Models for Disease Prediction
• SCALE: Selective Resource Allocation for Overcoming Performance Bottlenecks in Mathematical Test-time Scaling
• Think Before You Prune: Selective Self-Generated Calibration for Pruning Large Reasoning Models
• OpenMMReasoner: Pushing the Frontiers for Multimodal Reasoning with an Open and General Recipe
• SafeRBench: A Comprehensive Benchmark for Safety Assessment in Large Reasoning Models
• Reason-KE++: Aligning the Process, Not Just the Outcome, for Faithful LLM Knowledge Editing
• MetaGDPO: Alleviating Catastrophic Forgetting with Metacognitive Knowledge through Group Direct Preference Optimization
• Black-Box On-Policy Distillation of Large Language Models
• An Efficient Training Pipeline for Reasoning Graphical User Interface Agents
• Is It Truly Necessary to Process and Fit Minutes-Long Reference Videos for Personalized Talking Face Generation?
• Importance-Aware Data Selection for Efficient LLM Instruction Tuning
• Revisiting Entropy in Reinforcement Learning for Large Reasoning Models
• Why Less is More (Sometimes): A Theory of Data Curation
• Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberation
• Interaction as Intelligence Part II: Asynchronous Human-Agent Rollout for Long-Horizon Task Training
• VCORE: Variance-Controlled Optimization-based Reweighting for Chain-of-Thought Supervision
• BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuning
• Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning
• Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR
• Solving Physics Olympiad via Reinforcement Learning on Physics Simulators
• LimRank: Less is More for Reasoning-Intensive Information Reranking
• Boosting Accuracy and Efficiency of Budget Forcing in LLMs via Reinforcement Learning for Mathematical Reasoning
• What Defines Good Reasoning in LLMs? Dissecting Reasoning Steps with Multi-Aspect Evaluation
• Teaching Language Models to Reason with Tools
• DiffAdapt: Difficulty-Adaptive Reasoning for Token-Efficient LLM Inference
• A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning
• Mapping Post-Training Forgetting in Language Models at Scale
• QueST: Incentivizing LLMs to Generate Difficult Problems
• GroundedPRM: Tree-Guided and Fidelity-Aware Process Reward Modeling for Step-Level Reasoning
• Budget-aware Test-time Scaling via Discriminative Verification
• CoT-Evo: Evolutionary Distillation of Chain-of-Thought for Scientific Reasoning
• Demystifying Reinforcement Learning in Agentic Reasoning
• Parallel Scaling Law: Unveiling Reasoning Generalization through A Cross-Linguistic Perspective
• LLM Knowledge is Brittle: Truthfulness Representations Rely on Superficial Resemblance
• One Token Embedding Is Enough to Deadlock Your Large Reasoning Model
• Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead
• Prompting Test-Time Scaling Is A Strong LLM Reasoning Data Augmentation
• Mitigating Overthinking through Reasoning Shaping
• Logit Arithmetic Elicits Long Reasoning Capabilities Without Training
• All Code, No Thought: Current Language Models Struggle to Reason in Ciphered Language
• The Idola Tribus of AI: Large Language Models tend to perceive order where none exists
• ARES: Multimodal Adaptive Reasoning via Difficulty-Aware Token-Level Entropy Shaping
• Memory Retrieval and Consolidation in Large Language Models through Function Tokens
• Pragyaan: Designing and Curating High-Quality Cultural Post-Training Datasets for Indian Languages
• Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation
• Executable Counterfactuals: Improving LLMs' Causal Reasoning Through Code
• AsyncSpade: Efficient Test-Time Scaling with Asynchronous Sparse Decoding
• Off-Trajectory Reasoning: Can LLMs Collaborate on Reasoning Trajectory?
• TaTToo: Tool-Grounded Thinking PRM for Test-Time Scaling in Tabular Reasoning
• Influence Functions for Efficient Data Selection in Reasoning
• The Valley of Code Reasoning: Scaling Knowledge Distillation of Large Language Models
• Detecting Distillation Data from Reasoning Models
• Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought
• Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning
• Distilling Reasoning into Student LLMs: Local Naturalness for Selecting Teacher Data
• Don't Pass@k: A Bayesian Framework for Large Language Model Evaluation
• On Predictability of Reinforcement Learning Dynamics for Large Language Models
• A Tale of LLMs and Induced Small Proxies: Scalable Small Language Models for Knowledge Mining
• Prompt Curriculum Learning for Efficient LLM Post-Training
• DecepChain: Inducing Deceptive Reasoning in Large Language Models
• On The Fragility of Benchmark Contamination Detection in Reasoning Models
• Go with Your Gut: Scaling Confidence for Autoregressive Image Generation
• Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners
• OPPO: Accelerating PPO-based RLHF via Pipeline Overlap
• Pushing LLMs to Their Logical Reasoning Bound: The Role of Data Reasoning Intensity
• ReasonCACHE: Teaching LLMs To Reason Without Weight Updates
• Winning the Pruning Gamble: A Unified Approach to Joint Sample and Token Pruning for Efficient Supervised Fine-Tuning
• Poivre: Self-Refining Visual Pointing with Reinforcement Learning
• Beyond English-Centric Training: How Reinforcement Learning Improves Cross-Lingual Reasoning in LLMs
• Timber: Training-free Instruct Model Refining with Base via Effective Rank
• NanoFlux: Adversarial Dual-LLM Evaluation and Distillation For Multi-Domain Reasoning
• Best-of-∞ -- Asymptotic Performance of Test-Time Compute
• RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs
• The Conductor and the Engine: A Path Towards Co-Designed Reasoning
• Kalman Meets Curriculum: Efficient Dynamic Prompt Selection for Adaptive RL Finetuning
• Beyond the Best Teacher: Expanding and Compressing the Reasoning Solution Manifold
• Proximal Supervised Fine-Tuning
• CogAtom: From Cognitive Atoms to Olympiad-level Mathematical Reasoning in Large Language Models
• LIMI: Less is More for Agency
• Analyzing the Effects of Supervised Fine-Tuning on Model Knowledge from Token and Parameter Levels
• Meta-R1: Empowering Large Reasoning Models with Metacognition
• When Inverse Data Outperforms: Exploring the Pitfalls of Mixed Data in Multi-Stage Fine-Tuning
• BudgetThinker: Empowering Budget-aware LLM Reasoning with Control Tokens
• Metacognitive Reuse: Turning Recurring LLM Reasoning Into Concise Behaviors
• Reasoned Safety Alignment: Ensuring Jailbreak Defense via Answer-Then-Check
• Merge-of-Thought Distillation
• Small Vectors, Big Effects: A Mechanistic Study of RL-Induced Reasoning via Steering Vectors
• LAMDAS: LLM as an Implicit Classifier for Domain-specific Data Selection
• Implicit Reasoning in Large Language Models: A Comprehensive Survey
• VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use
• ParaThinker: Native Parallel Thinking as a New Paradigm to Scale LLM Test-time Compute
• MedGR²: Breaking the Data Barrier for Medical Reasoning via Generative Reward Learning
• SLIM: Subtrajectory-Level Elimination for More Effective Reasoning
• CAC-CoT: Connector-Aware Compact Chain-of-Thought for Efficient Reasoning Data Synthesis Across Dual-System Cognitive Tasks
• Fin-PRM: A Domain-Specialized Process Reward Model for Financial Reasoning in Large Language Models
• Deep Think with Confidence
• Input-Time Scaling
• G²RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidance
• User-Assistant Bias in LLMs
• On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting
• Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generation
• We-Math 2.0: A Versatile MathBook System for Incentivizing Visual Mathematical Reasoning
• Promoting Efficient Reasoning with Verifiable Stepwise Reward
• Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning
• MEML-GRPO: Heterogeneous Multi-Expert Mutual Learning for RLVR Advancement
• PrLM: Learning Explicit Reasoning for Personalized RAG via Contrastive Reward Optimization
• Sample-efficient LLM Optimization with Reset Replay
• Shuffle-R1: Efficient RL framework for Multimodal Large Language Models via Data-centric Dynamic Shuffle
• InfiAlign: A Scalable and Sample-Efficient Framework for Aligning LLMs to Enhance Reasoning Capabilities
• Light-IF: Endowing LLMs with Generalizable Reasoning via Preview and Self-Checking for Complex Instruction Following
• Towards Efficient Medical Reasoning with Minimal Fine-Tuning Data
• CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning tasks

Discussions

• LIMO: Less Is More for Reasoning [hn, 389 points, 128 comments]
• 7/7 This could change how we train AI models - shifting focus from data quantity to quality and changing how we think about AI learning and reasoning. arxiv.org/pdf/2502.03387 [bsky, 14 points, 1 comments]
• LIMO is a good example of this: they managed to get competitive reasoning benchmark scores with small models AND with <1000 samples (arxiv.org/abs/2502.03387) [bsky, 4 points, 0 comments]
• LIMO: Less Is More for Reasoning [hn, 2 points, 0 comments]
• Reading another paper arguing little data suffices to extract the skills already developed in pre-training. LIMO arxiv.org/abs/2502.03387. First sip but, bit self-aggrandizing? Scrolled to see model ( [bsky, 1 points, 1 comments]
• "LIMO: Less Is More for Reasoning" Some people think that using fewer, smarter examples can help teach math to computers. It's like getting better answers with less work. Article | Discussion [bsky, 0 points, 0 comments]
• https://bsky.app/profile/hackernews.com.web.brid.gy/post/3lhrseeagrr72 [bsky, 0 points, 0 comments]
• LIMO: Less Is More for Reasoning [bsky, 0 points, 0 comments]
• LIMO: Less Is More for Reasoning #HackerNews arxiv.org/abs/... [bsky, 0 points, 0 comments]
• LIMO: Less Is More for Reasoning https://arxiv.org/abs/2502.03387 https://news.ycombinator.com/item?id=42991676 [bsky, 0 points, 0 comments]
• LIMO: Less Is More for Reasoning https://arxiv.org/abs/2502.03387 arxiv.org [bsky, 0 points, 0 comments]
• LIMO: Less Is More for Reasoning https://arxiv.org/abs/2502.03387 [bsky, 0 points, 0 comments]
• LIMO: Less Is More for Reasoning https://arxiv.org/abs/2502.03387 [comments] [84 points] [bsky, 0 points, 0 comments]
• arxiv.org/abs/2502.03387 [bsky, 0 points, 0 comments]
• LIMO: Less Is More for Reasoning https://arxiv.org/abs/2502.03387 (https://news.ycombinator.com/item?id=42991676) [bsky, 0 points, 0 comments]
• Simplifying reasoning boosts decision-making efficiency. 🤖 #ai LIMO: Less Is More for Reasoning [bsky, 0 points, 0 comments]
• arxiv.org/abs/2502.03387 mhhhhhh [bsky, 0 points, 0 comments]
• LIMO: Less Is More for Reasoning https://arxiv.org/abs/2502.03387 (http://news.ycombinator.com/item?id=42991676) [bsky, 0 points, 0 comments]
• LIMO: Less Is More for Reasoning https://arxiv.org/abs/2502.03387 (http://news.ycombinator.com/item?id=42991676) [bsky, 0 points, 0 comments]
• LIMO: Less Is More for Reasoning (arxiv.org) Main Link | Discussion [bsky, 0 points, 0 comments]
• LIMO Hypothesis: Suggests that reasoning capabilities in pre-trained models emerge with precisely crafted cognitive templates rather than massive datasets. arxiv.org/abs/2502.03387 [bsky, 0 points, 0 comments]
• LIMO: Less Is More for Reasoning view on hacker news [bsky, 0 points, 0 comments]
• LIMO: Less is More for Reasoning arxiv.org/abs/2502.03387 사람을 교육하는 것도 많은 내용이 아니라 핵심적이고 잘 정리된 내용이 더 효과적인 것과 비슷하다. [bsky, 0 points, 0 comments]
• LIMO: Less Is More for Reasoning https://arxiv.org/abs/2502.03387 [bsky, 0 points, 0 comments]
• LIMO: Less Is More for Reasoning https://arxiv.org/abs/2502.03387 (https://news.ycombinator.com/item?id=42991676) [bsky, 0 points, 0 comments]
• LIMO: Less Is More for Reasoning https://arxiv.org/abs/2502.03387 (https://news.ycombinator.com/item?id=42991676) [bsky, 0 points, 0 comments]
• Paper link https://buff.ly/3QaGOiw The approach could revolutionize how we develop specialized AI models. Quality over quantity might be the future of AI training. #AI #MachineLearning #DeepLearni [bsky, 0 points, 0 comments]
• arxiv.org/abs/2502.03387 [bsky, 0 points, 1 comments]

Related

• When Does Reasoning Matter? A Controlled Study of Reasoning's Contribution to Model Performance
• Specializing Smaller Language Models towards Multi-Step Reasoning
• LIMOPro: Reasoning Refinement for Efficient and Effective Test-time Scaling
• Scaling Relationship on Learning Mathematical Reasoning with Large Language Models
• MobileLLM-R1: Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes