Reconstructive Visual Instruction Tuning

2024/10/12 by Haochen Wang, Wang, Haochen, Anlin Zheng +11 · 25 citations
Neuroscience · Social Sciences · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Education and Technology Integration #FOS: Computer and information sciences #Machine Learning (cs.LG) #Tactile and Sensory Interactions

paper · pdf · doi:10.48550/arxiv.2410.09575

openalex publication_date 2024/10/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper introduces reconstructive visual instruction tuning (ROSS), a family of Large Multimodal Models (LMMs) that exploit vision-centric supervision signals. In contrast to conventional visual instruction tuning approaches that exclusively supervise text outputs, ROSS prompts LMMs to supervise visual outputs via reconstructing input images. By doing so, it capitalizes on the inherent richness and detail present within input images themselves, which are often lost in pure text supervision. However, producing meaningful feedback from natural images is challenging due to the heavy spatial redundancy of visual signals. To address this issue, ROSS employs a denoising objective to reconstruct latent representations of input images, avoiding directly regressing exact raw RGB values. This intrinsic activation design inherently encourages LMMs to maintain image detail, thereby enhancing their fine-grained comprehension capabilities and reducing hallucinations. Empirically, ROSS consistently brings significant improvements across different visual encoders and language models. In comparison with extrinsic assistance state-of-the-art alternatives that aggregate multiple visual experts, ROSS delivers competitive performance with a single SigLIP visual encoder, demonstrating the efficacy of our vision-centric supervision tailored for visual outputs.

Cited by

• Sketch-in-Latents: Eliciting Unified Reasoning in MLLMs
• Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Models
• Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models
• TUNA: Taming Unified Visual Representations for Native Unified Multimodal Models
• MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal LLMs
• Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidence
• Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMs
• MT-Video-Bench: A Holistic Video Understanding Benchmark for Evaluating Multimodal LLMs in Multi-Turn Dialogues
• Vision-Centric Activation and Coordination for Multimodal Large Language Models
• SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Models
• Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Model
• DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Driving
• Unified Multimodal Model as Auto-Encoder
• Visual Representation Alignment for Multimodal Large Language Models
• Reconstruction Alignment Improves Unified Multimodal Models
• Autoregressive Semantic Visual Reconstruction Helps VLMs Understand Better
• Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodology
• Hita: Holistic Tokenizer for Autoregressive Image Generation
• VGR: Visual Grounded Reasoning
• un²CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIP
• Exploring The Visual Feature Space for Multimodal Neural Decoding
• Visual Instruction Bottleneck Tuning
• VISTA: Enhancing Vision-Text Alignment in MLLMs via Cross-Modal Mutual Information Maximization
• Prioritizing Image-Related Tokens Enhances Vision-Language Pre-Training
• The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformer

Related

• Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning
• Visual Instruction Tuning towards General-Purpose Multimodal Model: A Survey
• Learning to Instruct for Visual Instruction Tuning
• Do we Really Need Visual Instructions? Towards Visual Instruction-Free Fine-tuning for Large Vision-Language Models
• VIGC: Visual Instruction Generation and Correction