2024/12/11 by Michal Shlapentokh-Rothman, Yu-Xiong Wang, Shlapentokh-Rothman, Michal +3
Computer Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Model-Driven Software Engineering Techniques #Open Education and E-Learning
paper · pdf · doi:10.48550/arxiv.2412.08564
openalex publication_date 2024/12/11 · openalex created_date 2024/12/13 · openalex updated_date 2026/07/28
Adapting visual programming or prompting large language models (LLMs) to generate executable code for visual tasks like visual question answering (VQA) for specialized tasks or domains remains challenging due to high annotation and inference costs. We propose a low-cost visual program distillation method that can be used for models with at most 1 billion parameters and requires no human-generated program annotations. We achieve this through synthetic data augmentation based on decoupling programs into higher-level skills, called templates, and their corresponding arguments. Experimental results show that, with a relatively small amount of question/answer data, small language models can generate high-quality specialized visual programs with the added benefit of much faster inference