To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuning

2023/11/13 by Junke Wang, Lingchen Meng, Wang, Junke +9 · 19 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2311.07574

openalex publication_date 2023/11/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Existing visual instruction tuning methods typically prompt large language models with textual descriptions to generate instruction-following data. Despite the promising performance achieved, these descriptions are derived from image annotations, which are oftentimes coarse-grained. Furthermore, the instructions might even contradict the visual content without observing the entire visual context. To address this challenge, we introduce a fine-grained visual instruction dataset, LVIS-Instruct4V, which contains 220K visually aligned and context-aware instructions produced by prompting the powerful GPT-4V with images from LVIS. Through experimental validation and case studies, we demonstrate that high-quality visual instructional data could improve the performance of LLaVA-1.5, a state-of-the-art large multimodal model, across a wide spectrum of benchmarks by clear margins. Notably, by simply replacing the LLaVA-Instruct with our LVIS-Instruct4V, we achieve better results than LLaVA on most challenging LMM benchmarks, e.g., LLaVA^w (76.7 vs. 70.7) and MM-Vet (40.2 vs. 35.4). We release our data and model at https://github.com/X2FD/LVIS-INSTRUCT4V.

Cited by

• LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sight
• Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Experts
• FineVision: Open Data Is All You Need
• From Pixels to Words -- Towards Native Vision-Language Primitives at Scale
• Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Models
• MATRIX: Multimodal Agent Tuning for Robust Tool-Use Reasoning
• Dirichlet-Prior Shaping: Guiding Expert Specialization in Upcycled MoEs
• MindVL: Towards Efficient and Effective Training of Multimodal Large Language Models on Ascend NPUs
• Improving Large Vision and Language Models by Learning from a Panel of Peers
• UItron: Foundational GUI Agent with Advanced Perception and Planning
• Metadata Management for AI-Augmented Data Workflows
• VER-Bench: Evaluating MLLMs on Reasoning with Fine-Grained Visual Evidence
• See Different, Think Better: Visual Variations Mitigating Hallucinations in LVLMs
• Mono-InternVL-1.5: Towards Cheaper and Faster Monolithic Multimodal Large Language Models
• Advancing Multimodal LLMs by Large-Scale 3D Visual Instruction Dataset Generation
• AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding
• MoTE: Mixture of Ternary Experts for Memory-efficient Large Multimodal Models
• Pisces: An Auto-regressive Foundation Model for Image Understanding and Generation
• SMAR: Soft Modality-Aware Routing Strategy for MoE-based Multimodal Large Language Models Preserving Language Capabilities

Related

• Visual Instruction Tuning
• Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning
• Learning to Instruct for Visual Instruction Tuning
• Generative Visual Instruction Tuning
• VIGC: Visual Instruction Generation and Correction