S*: Test Time Scaling for Code Generation

2025/02/20 by Li, Dacheng, Cao, Shiyi, Cao, Chengkun +6 · 24 citations
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2502.14382

Abstract

Increasing test-time compute for LLMs shows promise across domains but remains underexplored in code generation, despite extensive study in math. In this paper, we propose S*, the first hybrid test-time scaling framework that substantially improves the coverage and selection accuracy of generated code. S* extends the existing parallel scaling paradigm with sequential scaling to push performance boundaries. It further leverages a novel selection mechanism that adaptively generates distinguishing inputs for pairwise comparison, combined with execution-grounded information to robustly identify correct solutions. We evaluate across 12 Large Language Models and Large Reasoning Model and show: (1) S* consistently improves performance across model families and sizes, enabling a 3B model to outperform GPT-4o-mini; (2) S* enables non-reasoning models to surpass reasoning models - GPT-4o-mini with S* outperforms o1-preview by 3.7% on LiveCodeBench; (3) S* further boosts state-of-the-art reasoning models - DeepSeek-R1-Distill-Qwen-32B with S* achieves 85.7% on LiveCodeBench, approaching o1 (high) at 88.5%. Code will be available under https://github.com/NovaSky-AI/SkyThought.

Cited by

• Evaluating Code Reasoning Abilities of Large Language Models Under Real-World Settings
• Rethinking Test Time Scaling for Flow-Matching Generative Models
• Is the Cure Still Worse Than the Disease? Test Overfitting by LLMs in Automated Program Repair
• InfCode-C++: Intent-Guided Semantic Retrieval and AST-Structured Search for C++ Issue Resolution
• Reasoning Planning for Language Models
• SELF-REDRAFT: Eliciting Intrinsic Exploration-Exploitation Balance in Test-Time Scaling for Code Generation
• RETuning: Upgrading Inference-Time Scaling for Stock Movement Prediction with Large Language Models
• LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?
• Automated Discovery of Test Oracles for Database Management Systems Using LLMs
• Test-Time Scaling of Reasoning Models for Machine Translation
• ARISE: An Adaptive Resolution-Aware Metric for Test-Time Scaling Evaluation in Large Reasoning Models
• CodeChemist: Functional Knowledge Transfer for Low-Resource Code Generation via Test-Time Scaling
• R-Log: Incentivizing Log Analysis Capability in LLMs via Reasoning-based Reinforcement Learning
• Adaptive Test-Time Reasoning via Reward-Guided Dual-Phase Search
• ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory
• ATTS: Asynchronous Test-Time Scaling via Conformal Prediction
• VerilogMonkey: Exploring Parallel Scaling for Automated Verilog Code Generation with LLMs
• LTA-thinker: Latent Thought-Augmented Training Framework for Large Language Models on Complex Reasoning
• Routing Distilled Knowledge via Mixture of LoRA Experts for Large Language Model based Bundle Generation
• app.build: A Production Framework for Scaling Agentic Prompt-to-App Generation with Environment Scaffolding
• SSRL: Self-Search Reinforcement Learning
• MEML-GRPO: Heterogeneous Multi-Expert Mutual Learning for RLVR Advancement
• Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scaling
• Diversity-Enhanced Reasoning for Subjective Questions

Related

• SCALE: Selective Resource Allocation for Overcoming Performance Bottlenecks in Mathematical Test-time Scaling
• SETS: Leveraging Self-Verification and Self-Correction for Improved Test-Time Scaling
• Are We Scaling the Right Thing? A System Perspective on Test-Time Scaling
• Revisiting Test-Time Scaling: A Survey and a Diversity-Aware Method for Efficient Reasoning
• Z1: Efficient Test-time Scaling with Code