Trust the Model: Compact VLMs as In-Context Judges for Image-Text Data Quality

2025/07/27 by Daulet Toibazar, Kesen Wang, Toibazar, Daulet +9
Computer Science · #Advanced Image and Video Retrieval Techniques #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Data modeling #Data quality #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Filter (signal processing) #Filtration (mathematics) #Multimodal Machine Learning Applications #Quality (philosophy) #Training (meteorology) #Training set

paper · pdf · doi:10.48550/arxiv.2507.20156

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/07/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Vision-language models (VLMs) extend the conventional large language models by integrating visual data, enabling richer multimodal reasoning and significantly broadens the practical applications of AI. However, including visual inputs also brings new challenges in maintaining data quality. Empirical evidence consistently shows that carefully curated and representative training examples often yield superior results compared to simply increasing the quantity of data. Inspired by this observation, we introduce a streamlined data filtration framework that employs a compact VLM, fine-tuned on a high-quality image-caption annotated dataset. This model effectively evaluates and filters potential training samples based on caption and image quality and alignment. Unlike previous approaches, which typically add auxiliary filtration modules on top of existing full-scale VLMs, our method exclusively utilizes the inherent evaluative capability of a purpose-built small VLM. This strategy eliminates the need for extra modules and reduces training overhead. Our lightweight model efficiently filters out inaccurate, noisy web data, improving image-text alignment and caption linguistic fluency. Experimental results show that datasets underwent high-precision filtration using our compact VLM perform on par with, or even surpass, larger and noisier datasets gathered through high-volume web crawling. Thus, our method provides a lightweight yet robust solution for building high-quality vision-language training corpora. Availability and implementation: Our compact VLM filtration model, training data, utility scripts, and Supplementary data (Appendices) are freely available at https://github.com/daulettoibazar/Compact_VLM_Filter.

Citations

• Global Semantic-Guided Sub-image Feature Weight Allocation in High-Resolution Large Vision-Language Models
• Towards Efficient Visual-Language Alignment of the Q-Former for Visual Reasoning Tasks
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• LLaVA-OneVision: Easy Visual Task Transfer
• What If We Recaption Billions of Web Images with LLaMA-3?
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
• Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selection
• A Survey on Hallucination in Large Vision-Language Models
• mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration
• MM-LIMA: Less Is More for Alignment in Multi-Modal Datasets
• AlpaGasus: Training A Better Alpaca with Fewer Data
• OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents
• LIMA: Less Is More for Alignment
• MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
• Visual Instruction Tuning
• On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?
• Training language models to follow instructions with human feedback
• A Frustratingly Simple Approach for End-to-End Image Captioning
• Learning Transferable Visual Models From Natural Language Supervision
• Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• SciPy 1.0--Fundamental Algorithms for Scientific Computing in Python
• AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding

Related

• Filter & Align: Leveraging Human Knowledge to Curate Image-Text Data
• Better Reasoning with Less Data: Enhancing VLMs Through Unified Modality Scoring
• MiniVLM: A Smaller and Faster Vision-Language Model
• SynthVLM: Towards High-Quality and Efficient Synthesis of Image-Caption Datasets for Vision-Language Models
• BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation