CodeChemist: Functional Knowledge Transfer for Low-Resource Code Generation via Test-Time Scaling

2025/10/01 by Kaixin Wang, T. Li, Wang, Kaixin +14
Computer Science · #Educational Technology and Assessment #FOS: Computer and information sciences #Software Engineering (cs.SE) #Software Testing and Debugging Techniques #Speech and dialogue systems

paper · pdf · doi:10.48550/arxiv.2510.00501

openalex publication_date 2025/10/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Code Large Language Models (CodeLLMs) have been widely adopted for Natural Language to Programming Language code generation, powering applications with large user bases. Their performance, however, varies sharply across programming languages (PLs) and is particularly suboptimal for low-resource PLs due to data scarcity, limiting their overall usability. In this work, we introduce CodeChemist, a simple yet effective, training-free test-time scaling framework that transfers the model's functional knowledge from high-resource to low-resource PLs via synthesized test cases, without relying on external models. Specifically, CodeChemist first applies multi-temperature hedged sampling to generate a pool of candidate solutions in the low-resource PL and synthesizes a set of test inputs. It then estimates uncertainty: when uncertainty is low, it selects the output via in-language majority voting; otherwise, it constructs cross-lingual I/O test oracles by executing high-resource reference programs and selects the candidate with the highest pass rate. Extensive experiments demonstrate that CodeChemist significantly outperforms existing test-time scaling methods, improving code generation for both low-resource PLs (e.g., Lua) and complex-syntax PLs (e.g., C++, Java) without retraining.

Citations

• Learning to Generate Unit Test via Adversarial Reinforcement Learning
• Agnostics: Learning to Code in Any Programming Language via Reinforcement with a Universal Learning Environment
• When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMs
• Exploring the Impact of Temperature on Large Language Models:Hot or Cold?
• Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agents
• S*: Test Time Scaling for Code Generation
• MCTS-Judge: Test-Time Scaling in LLM-as-a-Judge for Code Correctness Evaluation
• ACECODER: Acing Coder RL via Automated Test-Case Synthesis
• Enhancing Code Generation for Low-Resource Languages: No Silver Bullet
• CodeMonkeys: Scaling Test-Time Compute for Software Engineering
• The Limits of Inference Scaling Through Resampling
• Preference Optimization for Reasoning with Pseudo Feedback
• GPT-4o System Card
• Bridge-Coder: Unlocking LLMs' Potential to Overcome Language Gaps in Low-Resource Code
• Qwen2.5-Coder Technical Report
• Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
• The Llama 3 Herd of Models
• Uncovering LLM-Generated Code: A Zero-Shot Synthetic Code Detector via Code Rewriting
• The Larger the Better? Improved LLM Code-Generation via Budget Reallocation
• LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
• Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations
• SGLang: Efficient Execution of Structured Language Model Programs
• Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency
• Large Language Models for Software Engineering: A Systematic Literature Review
• Large Language Models for Software Engineering: A Systematic Literature Review
• Knowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMs
• The potential of LLMs for coding with low-resource and domain-specific programming languages
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Let's Verify Step by Step
• CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X
• MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation
• CodeT: Code Generation with Generated Tests
• On the Transferability of Pre-trained Language Models for Low-Resource Programming Languages
• Self-Consistency Improves Chain of Thought Reasoning in Language Models
• Leveraging Automated Unit Tests for Unsupervised Code Translation
• Program Synthesis with Large Language Models
• Evaluating Large Language Models Trained on Code
• Deep reinforcement learning from human preferences

Related

• Knowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMs
• CodeT: Code Generation with Generated Tests
• CodeACT: Code Adaptive Compute-efficient Tuning Framework for Code LLMs
• CodeIF: Benchmarking the Instruction-Following Capabilities of Large Language Models for Code Generation
• Enhancing Code Generation for Low-Resource Languages: No Silver Bullet