Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling

2025/02/10 by Runze Liu, Liu, Runze, Junqi Gao +13 · 4 voices · 36 citations
Computer Science · #Digital Rights Management and Security #cs.CL

paper · pdf · doi:10.48550/arxiv.2502.06703

openalex publication_date 2025/02/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Test-Time Scaling (TTS) is an important method for improving the performance of Large Language Models (LLMs) by using additional computation during the inference phase. However, current studies do not systematically analyze how policy models, Process Reward Models (PRMs), and problem difficulty influence TTS. This lack of analysis limits the understanding and practical use of TTS methods. In this paper, we focus on two core questions: (1) What is the optimal approach to scale test-time computation across different policy models, PRMs, and problem difficulty levels? (2) To what extent can extended computation improve the performance of LLMs on complex tasks, and can smaller language models outperform larger ones through this approach? Through comprehensive experiments on MATH-500 and challenging AIME24 tasks, we have the following observations: (1) The compute-optimal TTS strategy is highly dependent on the choice of policy model, PRM, and problem difficulty. (2) With our compute-optimal TTS strategy, extremely small policy models can outperform larger models. For example, a 1B LLM can exceed a 405B LLM on MATH-500. Moreover, on both MATH-500 and AIME24, a 0.5B LLM outperforms GPT-4o, a 3B LLM surpasses a 405B LLM, and a 7B LLM beats o1 and DeepSeek-R1, while with higher inference efficiency. These findings show the significance of adapting TTS strategies to the specific characteristics of each task and model and indicate that TTS is a promising approach for enhancing the reasoning abilities of LLMs.

Cited by

• Scaling Unverifiable Rewards: A Case Study on Visual Insights
• Demystifying LLM-as-a-Judge: Analytically Tractable Model for Inference-Time Scaling
• Let's (not) just put things in Context: Test-Time Training for Long-Context LLMs
• No Labels, No Problem: Training Visual Reasoners with Multimodal Verifiers
• Think in Parallel, Answer as One: Logit Averaging for Open-Ended Reasoning
• ITS3D: Inference-Time Scaling for Text-Guided 3D Diffusion Models
• SPINE: Token-Selective Test-Time Reinforcement Learning with Entropy-Band Regularization
• Generalizing Test-time Compute-optimal Scaling as an Optimizable Graph
• MAS-ProVe: Understanding the Process Verification of Multi-Agent Systems
• CarBoN: Calibrated Best-of-N Sampling Improves Test-time Reasoning
• GroundedPRM: Tree-Guided and Fidelity-Aware Process Reward Modeling for Step-Level Reasoning
• Inference-Time Search using Side Information for Diffusion-based Image Reconstruction
• RoVer: Robot Reward Model as Test-Time Verifier for Vision-Language-Action Model
• RLFR: Extending Reinforcement Learning for LLMs with Flow Environment
• On the Role of Temperature Sampling in Test-Time Scaling
• TaTToo: Tool-Grounded Thinking PRM for Test-Time Scaling in Tabular Reasoning
• Thinking on the Fly: Test-Time Reasoning Enhancement via Latent Thought Policy Optimization
• Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Models
• Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewards
• Scaling LLM Test-Time Compute with Mobile NPU on Smartphones
• From Harm to Help: Turning Reasoning In-Context Demos into Assets for Reasoning LMs
• Dynamic Experts Search: Enhancing Reasoning in Mixture-of-Experts LLMs at Test Time
• Variation in Verification: Understanding Verification Dynamics in Large Language Models
• LATTS: Locally Adaptive Test-Time Scaling
• Explicit Reasoning Makes Better Judges: A Systematic Study on Accuracy, Efficiency, and Robustness
• Instance-Wise Adaptive Sampling for Dataset Construction in Approximating Inverse Problem Solutions
• Democratizing Agentic AI with Fast Test-Time Scaling on the Edge
• The Complexity Trap: Simple Observation Masking Is as Efficient as LLM Summarization for Agent Context Management
• Test-Time Scaling Strategies for Generative Retrieval in Multimodal Conversational Recommendations
• Fin-PRM: A Domain-Specialized Process Reward Model for Financial Reasoning in Large Language Models
• Retrieval-augmented reasoning with lean language models
• ETTRL: Balancing Exploration and Exploitation in LLM Test-Time Reinforcement Learning Via Entropy Mechanism
• SSRL: Self-Search Reinforcement Learning
• Causal Reward Adjustment: Mitigating Reward Hacking in External Reasoning via Backdoor Correction
• Diversity-Enhanced Reasoning for Subjective Questions
• AgentTTS: Large Language Model Agent for Test-time Compute-optimal Scaling Strategy in Complex Tasks

Discussions

• Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling [lobsters, 4 points, 0 comments]
• Good paper, I'll have to add this to my post on AI scaling. More evidence that frontier models will get ~100x smaller and that test-time scaling gains asymptote hard. arxiv.org/abs/2502.06703 [bsky, 1 points, 1 comments]
• Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling https://lobste.rs/s/xzrfy0 #ai [bsky, 0 points, 0 comments]
• Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling arxiv.org/abs/2502.06703 [bsky, 0 points, 0 comments]

Related

• The Art of Scaling Test-Time Compute for Large Language Models
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
• Are We Scaling the Right Thing? A System Perspective on Test-Time Scaling
• Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling
• Test-time Scaling of LLMs: A Survey from A Subproblem Structure Perspective