SynthVLM: Towards High-Quality and Efficient Synthesis of Image-Caption Datasets for Vision-Language Models

2024/07/30 by Zheng Liu, Liu, Zheng, Hao Liang +11 · 10 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2407.20756

openalex publication_date 2024/07/30 · openalex created_date 2024/08/01 · openalex updated_date 2026/07/28

Abstract

Vision-Language Models (VLMs) have recently emerged, demonstrating remarkable vision-understanding capabilities. However, training these models requires large-scale datasets, which brings challenges related to efficiency, effectiveness, and quality of web data. In this paper, we introduce SynthVLM, a new data synthesis and curation method for generating image-caption pairs. Unlike traditional methods, where captions are generated from images, SynthVLM utilizes advanced diffusion models and high-quality captions to synthesize and select images from text captions, thereby creating precisely aligned image-text pairs. We further introduce SynthVLM-100K, a high-quality dataset consisting of 100K curated and synthesized image-caption pairs. In both model and human evaluations, SynthVLM-100K outperforms traditional real-world datasets. Leveraging this dataset, we develop a new family of multimodal large language models (MLLMs), SynthVLM-7B and SynthVLM-13B, which achieve state-of-the-art (SOTA) performance on various vision question-answering (VQA) tasks. Notably, our models outperform LLaVA across most metrics with only 18% pretrain data. Furthermore, SynthVLM-7B and SynthVLM-13B attain SOTA performance on the MMLU benchmark, demonstrating that the high-quality SynthVLM-100K dataset preserves language abilities.

Cited by

• Differences That Matter: Auditing Models for Capability Gap Discovery and Rectification
• RecTok: Reconstruction Distillation along Rectified Flow
• The Unseen Bias: How Norm Discrepancy in Pre-Norm MLLMs Leads to Visual Information Loss
• DEJIMA: A Novel Large-scale Japanese Dataset for Image Captioning and Visual Question Answering
• Physics-Based Benchmarking Metrics for Multimodal Synthetic Images
• Bridging the behavior-neural gap: A multimodal AI reveals the brain's geometry of emotion more accurately than human self-reports
• Object Detection with Multimodal Large Vision-Language Models: An In-depth Review
• LoRA-Loop: Closing the Synthetic Replay Cycle for Continual VLM Learning
• Effective Training Data Synthesis for Improving MLLM Chart Understanding
• Advancing Multimodal LLMs by Large-Scale 3D Visual Instruction Dataset Generation

Related

• Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
• Synth²: Boosting Visual-Language Models with Synthetic Captions and Image Embeddings
• VisCon-100K: Leveraging Contextual Web Data for Fine-tuning Vision Language Models
• Enhancing Vision-Language Pre-Training with Jointly Learned Questioner and Dense Captioner
• Unicorn: Text-Only Data Synthesis for Vision Language Model Training