S*: Test Time Scaling for Code Generation

2025/02/20 by Dacheng Li, Shiyi Cao, Li, Dacheng +15 · 1 voice · 46 citations
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Embedded Systems Design Techniques #FOS: Computer and information sciences #Machine Learning (cs.LG) #Real-time simulation and control systems #Software Testing and Debugging Techniques #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.2502.14382

openalex publication_date 2025/02/20 · arxiv published 2025/02/20 · arxiv updated 2025/02/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Increasing test-time compute for LLMs shows promise across domains but remains underexplored in code generation, despite extensive study in math. In this paper, we propose S*, the first hybrid test-time scaling framework that substantially improves the coverage and selection accuracy of generated code. S* extends the existing parallel scaling paradigm with sequential scaling to push performance boundaries. It further leverages a novel selection mechanism that adaptively generates distinguishing inputs for pairwise comparison, combined with execution-grounded information to robustly identify correct solutions. We evaluate across 12 Large Language Models and Large Reasoning Model and show: (1) S* consistently improves performance across model families and sizes, enabling a 3B model to outperform GPT-4o-mini; (2) S* enables non-reasoning models to surpass reasoning models - GPT-4o-mini with S* outperforms o1-preview by 3.7% on LiveCodeBench; (3) S* further boosts state-of-the-art reasoning models - DeepSeek-R1-Distill-Qwen-32B with S* achieves 85.7% on LiveCodeBench, approaching o1 (high) at 88.5%. Code will be available under https://github.com/NovaSky-AI/SkyThought.

Cited by

• Evaluating Code Reasoning Abilities of Large Language Models Under Real-World Settings
• Rethinking Test Time Scaling for Flow-Matching Generative Models
• Is the Cure Still Worse Than the Disease? Test Overfitting by LLMs in Automated Program Repair
• InfCode-C++: Intent-Guided Semantic Retrieval and AST-Structured Search for C++ Issue Resolution
• Reasoning Planning for Language Models
• SELF-REDRAFT: Eliciting Intrinsic Exploration-Exploitation Balance in Test-Time Scaling for Code Generation
• RETuning: Upgrading Inference-Time Scaling for Stock Movement Prediction with Large Language Models
• LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?
• Automated Discovery of Test Oracles for Database Management Systems Using LLMs
• Test-Time Scaling of Reasoning Models for Machine Translation
• ARISE: An Adaptive Resolution-Aware Metric for Test-Time Scaling Evaluation in Large Reasoning Models
• CodeChemist: Test-Time Scaling for Low-Resource Code Generation via Functional Knowledge Transfer
• R-Log: Incentivizing Log Analysis Capability in LLMs via Reasoning-based Reinforcement Learning
• Adaptive Test-Time Reasoning via Reward-Guided Dual-Phase Search
• ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory
• ATTS: Asynchronous Test-Time Scaling via Conformal Prediction
• VerilogMonkey: Exploring Parallel Scaling for Automated Verilog Code Generation with LLMs
• LTA-thinker: Latent Thought-Augmented Training Framework for Large Language Models on Complex Reasoning
• Routing Distilled Knowledge via Mixture of LoRA Experts for Large Language Model based Bundle Generation
• app.build: A Production Framework for Scaling Agentic Prompt-to-App Generation with Environment Scaffolding
• SSRL: Self-Search Reinforcement Learning
• MEML-GRPO: Heterogeneous Multi-Expert Mutual Learning for RLVR Advancement
• Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scaling
• Diversity-Enhanced Reasoning for Subjective Questions
• V₁: Unifying Generation and Self-Verification for Parallel Reasoners
• R-Stitch: Dynamic Trajectory Stitching for Efficient Reasoning
• Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Study
• RoboMonkey: Scaling Test-Time Sampling and Verification for Vision-Language-Action Models
• Code Reasoning for Software Engineering Tasks: A Survey and A Call to Action
• CP-Bench: Evaluating Large Language Models for Constraint Modelling
• SPRINT: Enabling Interleaved Planning and Parallelized Execution in Reasoning Models
• ScaleRTL: Scaling LLMs with Reasoning Data and Test-Time Compute for Accurate RTL Code Generation
• Co-Evolving LLM Coder and Unit Tester via Reinforcement Learning
• Generalizable LLM Learning of Graph Synthetic Data with Post-training Alignment
• Scaling Textual Gradients via Sampling-Based Momentum
• AlphaOne: Reasoning Models Thinking Slow and Fast at Test Time
• Co-PatcheR: Collaborative Software Patching with Component(s)-specific Small Reasoning Models
• First Finish Search: Efficient Test-Time Scaling in Large Language Models
• GeoGramBench: Benchmarking the Geometric Program Reasoning in Modern LLMs
• SSR: Speculative Parallel Scaling Reasoning in Test-time
• Towards Functional Correctness of Large Code Models with Selective Generation
• SoftCoT++: Test-Time Scaling with Soft Chain-of-Thought Reasoning
• Visual Verification Enables Inference-time Steering and Autonomous Policy Improvement
• EGSS: Entropy-guided Stepwise Scaling for Reliable Software Engineering
• Frontier AI's Impact on the Cybersecurity Landscape
• Learning Adaptive Parallel Reasoning with Language Models

Discussions

• test-time scalingをコード生成に使うという話。iterativeにdebuggingしてできた複数の生成コードに対して、あまり信頼のおけないLLMのテストコード生成と結果のクラスタリングによる一対比較によって、一番良いコードを選択する、っていう話かな。素のDeepSeekに対し20ポイントくらい性能向上してるのが興味深い。 arxiv.org/abs/2502.14382 [bsky, 0 points, 0 comments]

Related

• SCALE: Selective Resource Allocation for Overcoming Performance Bottlenecks in Mathematical Test-time Scaling
• SETS: Leveraging Self-Verification and Self-Correction for Improved Test-Time Scaling
• Are We Scaling the Right Thing? A System Perspective on Test-Time Scaling
• Revisiting Test-Time Scaling: A Survey and a Diversity-Aware Method for Efficient Reasoning
• Z1: Efficient Test-time Scaling with Code