Improving Multimodal Datasets with Image Captioning

2023/07/19 by Thao D. Nguyen, Samir Yitzhak Gadre, Nguyen, Thao +7 · 23 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Artificial intelligence #Benchmark (surveying) #Closed captioning #Computer Vision and Pattern Recognition (cs.CV) #Computer science #FOS: Computer and information sciences #Image (mathematics) #Information retrieval #Key (lock) #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Natural language processing #Noise (video) #Raw data

paper · pdf · doi:10.48550/arxiv.2307.10350

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2023/07/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06

Abstract

Massive web datasets play a key role in the success of large vision-language models like CLIP and Flamingo. However, the raw web data is noisy, and existing filtering methods to reduce noise often come at the expense of data diversity. Our work focuses on caption quality as one major source of noise, and studies how generated captions can increase the utility of web-scraped datapoints with nondescript text. Through exploring different mixing strategies for raw and generated captions, we outperform the best filtering method proposed by the DataComp benchmark by 2% on ImageNet and 4% on average across 38 tasks, given a candidate pool of 128M image-text pairs. Our best approach is also 2x better at Flickr and MS-COCO retrieval. We then analyze what makes synthetic captions an effective source of text supervision. In experimenting with different image captioning models, we also demonstrate that the performance of a model on standard image captioning benchmarks (e.g., NoCaps CIDEr) is not a reliable indicator of the utility of the captions it generates for multimodal training. Finally, our experiments with using generated captions at DataComp's large scale (1.28B image-text pairs) offer insights into the limitations of synthetic text, as well as the importance of image curation with increasing training data quantity. The synthetic captions used in our experiments are now available on HuggingFace.

Cited by

• Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning
• Understanding the Gain from Data Filtering in Multimodal Contrastive Learning
• Culture Affordance Atlas: Reconciling Object Diversity Through Functional Mapping
• Enhancing the Outcome Reward-based RL Training of MLLMs with Self-Consistency Sampling
• "It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models
• Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retention
• Can They Dixit? Yes they Can! Dixit as a Playground for Multimodal Language Model Capabilities
• Addressing the ID-Matching Challenge in Long Video Captioning
• Efficient Discriminative Joint Encoders for Large Scale Vision-Language Reranking
• Person-Centric Annotations of LAION-400M: Auditing Bias and Its Transfer to Models
• Steering Multimodal Large Language Models Decoding for Context-Aware Safety
• MobileCLIP2: Improving Multi-Modal Reinforced Training
• CapRecover: A Cross-Modality Feature Inversion Attack Framework on Vision Language Models
• HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Models
• SynC: Synthetic Image Caption Dataset Refinement with One-to-many Mapping for Zero-shot Image Captioning
• LoRA-Loop: Closing the Synthetic Replay Cycle for Continual VLM Learning
• Ambiguity-Aware and High-Order Relation Learning for Multi-Grained Image-Text Matching
• Visual Semantic Description Generation with MLLMs for Image-Text Matching
• Large Multi-modal Model Cartographic Map Comprehension for Textual Locality Georeferencing
• ReME: A Data-Centric Framework for Training-Free Open-Vocabulary Segmentation
• Chain-of-Talkers (CoTalk): Fast Human Annotation of Dense Image Captions
• Scale Can't Overcome Pragmatics: The Impact of Reporting Bias on Vision-Language Reasoning
• VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models

Related

• Image Captioning with Multi-Context Synthetic Data
• A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions
• VeCLIP: Improving CLIP Training via Visual-enriched Captions
• Fine-grained Image Captioning with CLIP Reward
• Text Data-Centric Image Captioning with Interactive Prompts