Structured Program Synthesis using LLMs: Results and Insights from the IPARC Challenge

2025/06/15 by Surana, Shraddha, Srinivasan, Ashwin, Bain, Michael · 1 citation
Computer Science · #Software Engineering Research #Teaching and Learning Programming #Model-Driven Software Engineering Techniques

paper · pdf · doi:10.48550/arxiv.2506.13820

Abstract

The IPARC Challenge, inspired by ARC, provides controlled program synthesis tasks over synthetic images to evaluate automatic program construction, focusing on sequence, selection, and iteration. This set of 600 tasks has resisted automated solutions. This paper presents a structured inductive programming approach with LLMs that successfully solves tasks across all IPARC categories. The controlled nature of IPARC reveals insights into LLM-based code generation, including the importance of prior structuring, LLMs' ability to aid structuring (requiring human refinement), the need to freeze correct code, the efficiency of code reuse, and how LLM-generated code can spark human creativity. These findings suggest valuable mechanisms for human-LLM collaboration in tackling complex program synthesis.

Citations

• Engineering Scientific Assistants using Interactive Structured Induction of Programs
• ARC Prize 2024: Technical Report
• DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
• Structured Chain-of-Thought Prompting for Code Generation
• Structured Chain-of-Thought Prompting for Code Generation
• A Model for Intelligible Interaction Between Agents That Predict and Explain
• Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• DeepCoder: Learning to Write Programs
• I.—COMPUTING MACHINERY AND INTELLIGENCE
• The Prompt Report: A Systematic Survey of Prompt Engineering Techniques

Cited by

• Copyright Protection for Large Language Models: A Survey of Methods, Challenges, and Trends

Related

• Combining LLM Code Generation with Formal Specifications and Reactive Program Synthesis
• CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive Program Synthesis
• LLM-Guided Compositional Program Synthesis
• Function-constrained Program Synthesis
• Renaissance of Literate Programming in the Era of LLMs: Enhancing LLM-Based Code Generation in Large-Scale Projects