2024/06/17 by Jefferson Hernandez, Hernandez, Jefferson, Ruben Villegas +3
Engineering · Mathematics · Social Sciences · #Cognitive and developmental aspects of mathematical skills #Computer Vision and Pattern Recognition (cs.CV) #Education and Technology Integration #FOS: Computer and information sciences #Spatial Cognition and Navigation
paper · pdf · doi:10.48550/arxiv.2406.11262
openalex publication_date 2024/06/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
We propose to use automatically generated instruction-following data to improve the zero-shot capabilities of a large multimodal model with additional support for generative and image editing tasks. We achieve this by curating a new multimodal instruction-following set using GPT-4V and existing datasets for image generation and editing. Using this instruction set and the existing LLaVA-Finetune instruction set for visual understanding tasks, we produce GenLLaVA, a Generative Large Language and Visual Assistant. GenLLaVA is built through a strategy that combines three types of large pretrained models through instruction finetuning: Mistral for language modeling, SigLIP for image-text matching, and StableDiffusion for text-to-image generation. Our model demonstrates visual understanding capabilities superior to LLaVA and additionally demonstrates competitive results with native multimodal models such as Unified-IO 2, paving the way for building advanced general-purpose visual assistants by effectively re-using existing multimodal models. We open-source our dataset, codebase, and model checkpoints to foster further research and application in this domain.