Learning to Reason in 13 Parameters

2026/02/04 by John X. Morris, Niloofar Mireshghallah, Mark Ibrahim +1 · 13 voices · 2 citations
#cs.LG

paper · pdf

Abstract

Recent research has shown that language models can learn to reason, often via reinforcement learning. Some work even trains low-rank parameterizations for reasoning, but conventional LoRA cannot scale below the model dimension. We question whether even rank=1 LoRA is necessary for learning to reason and propose TinyLoRA, a method for scaling low-rank adapters to sizes as small as one parameter. Within our new parameterization, we are able to train the 8B parameter size of Qwen2.5 to 91% accuracy on GSM8K with only 13 trained parameters in bf16 (26 total bytes). We find this trend holds in general: we are able to recover 90% of performance improvements while training 1000x fewer parameters across a suite of more difficult learning-to-reason benchmarks such as AIME, AMC, and MATH500. Notably, we are only able to achieve such strong performance with RL: models trained using SFT require 100-1000x larger updates to reach the same performance.

Citations

• LIMO: Less is More for Reasoning
• LoRA vs Full Fine-tuning: An Illusion of Equivalence
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• LoRA: Low-Rank Adaptation of Large Language Models
• Playing Atari with Six Neurons

Cited by

• Non-vacuous Generalization Bounds for Reinforcement Learning with Verifiable Rewards
• Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights
• Thinking Sparks!: Emergent Attention Heads in Reasoning Models During Post Training

Discussions

• TinyLoRA – Learning to Reason in 13 Parameters [hn, 234 points, 45 comments]
• Wonder how few weights we can get away with updating arxiv.org/abs/2602.04118 [bsky, 5 points, 1 comments]
• Learning to Reason in 13 Parameters [hn, 3 points, 0 comments]
• Learning to Reason in 13 Parameters [hn, 2 points, 0 comments]
• "Learning to Reason in 13 parameters" An interesting paper by Meta: arxiv.org/abs/2602.04118 Authors propose "TinyLoRA". They showed that Qwen 8B parameter model was able to achieve about 92% accuracy [bsky, 1 points, 1 comments]
• TinyLoRA – Learning to Reason in 13 Parameters https://arxiv.org/abs/2602.04118 https://news.ycombinator.com/item?id=47541733 [bsky, 0 points, 0 comments]
• TinyLoRA – Learning to Reason in 13 Parameters https://arxiv.org/abs/2602.04118 [bsky, 0 points, 0 comments]
• TinyLoRA – Learning to Reason in 13 Parameters https://arxiv.org/abs/2602.04118 (http://news.ycombinator.com/item?id=47541733) [bsky, 0 points, 0 comments]
• TinyLoRA – Learning to Reason in 13 Parameters https:// arxiv.org/abs/2602.04118 # arxiv [mastodon, 0 points, 0 comments]
• Meta's TinyLoRA achieves effective reasoning in language models with just 13 parameters, enabling compute and energy efficient model customization arxiv.org/abs/2602.04118 [bsky, 0 points, 0 comments]
• Learning to "Reason" in 13 Parameters arxiv.org/pdf/2602.04118 [bsky, 0 points, 0 comments]
• https://bsky.app/profile/hackernews.com.web.brid.gy/post/3mify4ib7xxv2 [bsky, 0 points, 0 comments]
• TinyLoRA – Learning to Reason in 13 Parameters #HackerNews https://arxiv.org/abs/2602.04118 [bsky, 0 points, 0 comments]

Related

• Tina: Tiny Reasoning Models via LoRA
• Rank-1 LoRAs Encode Interpretable Reasoning Signals
• LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters
• Patience Is The Key to Large Language Model Reasoning
• Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't