2025/07/20 by Noah van der Vleuten, van der Vleuten, Noah · 1 voice
Computer Science · #Artificial Intelligence (cs.AI) #Embedded Systems Design Techniques #FOS: Computer and information sciences #Parallel Computing and Optimization Techniques #Software Engineering (cs.SE) #Software Engineering Research #cs.AI #cs.SE
paper · pdf · doi:10.48550/arxiv.2507.15889
openalex publication_date 2025/07/20 · arxiv published 2025/07/20 · arxiv updated 2025/07/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Language models for program synthesis are usually trained and evaluated on programming competition datasets (MBPP, APPS). However, these datasets are limited in size and quality, while these language models are extremely data hungry. Additionally, the language models have a misaligned program synthesis process compared to humans. While humans iteratively develop code with the help of a compiler, most program synthesis models currently produce code in one go. To solve these issues, we introduce a bootstrapping algorithm for program synthesis, that supports teaching models how to repair. We show that bootstrapping consistently outperforms regular fine-tuning. Compared to other work, our bootstrapped model performs on par with fine-tuned models that are 68% larger. Notably, bootstrapping with repairing also improves non-repairing performance compared to regular bootstrapping during inference. However, on our models, repairing during inference is likely inferior to simply sampling the same number of solutions. Furthermore, we find that there are issues with the example test cases in the training portion of the APPS dataset that are valuable to the community, as many repairing and reinforcement learning methods rely on them.