LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training
2025/09/28 by Xiang An, An, Xiang, Yin Xie +43 · 1 voice · 46 citations
#cs.CV
paper · pdf · doi:10.48550/arxiv.2509.23661
Abstract
We present LLaVA-OneVision-1.5, a novel family of Large Multimodal Models (LMMs) that achieve state-of-the-art performance with significantly reduced computational and financial costs. Different from the existing works, LLaVA-OneVision-1.5 provides an open, efficient, and reproducible framework for building high-quality vision-language models entirely from scratch. The LLaVA-OneVision-1.5 release comprises three primary components: (1) Large-Scale Curated Datasets: We construct an 85M concept-balanced pretraining dataset LLaVA-OneVision-1.5-Mid-Traning and a meticulously curated 22M instruction dataset LLaVA-OneVision-1.5-Instruct. (2) Efficient Training Framework: We develop a complete end-to-end efficient training framework leveraging an offline parallel data packing strategy to facilitate the training of LLaVA-OneVision-1.5 within a 16,000 budget. (3) State-of-the-art Performance: Experimental results demonstrate that LLaVA-OneVision-1.5 yields exceptionally competitive performance across a broad range of downstream tasks. Specifically, LLaVA-OneVision-1.5-8B outperforms Qwen2.5-VL-7B on 18 of 27 benchmarks, and LLaVA-OneVision-1.5-4B surpasses Qwen2.5-VL-3B on all 27 benchmarks. (4) RL-based Post-training: We unlock the model's latent potential through a lightweight RL stage, effectively eliciting robust chain-of-thought reasoning to significantly boost performance on complex multimodal reasoning tasks.
Citations
Cited by
- Generative Semantic Multi-Object Tracking: A Large-Scale Benchmark and an MLLM-Driven Reasoning Framework
- VISTA-Bench: Do Vision-Language Models Really Understand Visualized Text as Well as Pure Text?
- HiMemVLN: Enhancing Reliability of Open-Source Zero-Shot Vision-and-Language Navigation with Hierarchical Memory System
- VIABench: A Comprehensive Video Benchmark Collected from Blind Individuals for Visual Impairment Assistance
- DrawingVQA: A Real-World Benchmark for Multi-Depth Visual-Textual Reasoning on Construction Drawings
- DecoupleMix: Decoupled Ratio Search and Convex Allocation for Scalable VLM Data Recipes
- Adversarial Style Optimization: Enhancing VLM Jailbreaks by GRPO-based Stylistic Triggers Optimization
- Robo-Dopamine: General Process Reward Modeling for High-Precision Robotic Manipulation
- Masking Teacher and Reinforcing Student for Distilling Vision-Language Models
- UltraViT: Latency-Optimized On-device Vision Encoder for Large Vision-Language Models
- GeoAnchor: Collaborative Reasoning via Latent Decomposition for 3D Spatial Understanding
- UniPercept: Towards Unified Perceptual-Level Image Understanding across Aesthetics, Quality, Structure, and Texture
- DeliveryBench: Can Agents Earn Profit in Real World?
- CASA: Cross-Attention over Self-Attention for Efficient Vision-Language Fusion
- M3-Verse: A "Spot the Difference" Challenge for Large Multimodal Models
- GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation
- JMMMU-Pro: Image-based Japanese Multi-discipline Multimodal Understanding Benchmark via Vibe Benchmark Construction
- HERBench: A Benchmark for Multi-Evidence Integration in Video Question Answering
- DentalGPT: Incentivizing Multimodal Complex Reasoning in Dentistry
- IF-Bench: Benchmarking and Enhancing MLLMs for Infrared Images with Generative Visual Prompting
- InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Models
- Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models
- RefBench-PRO: Perceptual and Reasoning Oriented Benchmark for Referring Expression Comprehension
- EMMA: Efficient Multimodal Understanding, Generation, and Editing with a Unified Architecture
- Towards Cross-View Point Correspondence in Vision-Language Models
- TUNA: Taming Unified Visual Representations for Native Unified Multimodal Models
- WeMMU: Enhanced Bridging of Vision-Language Models and Diffusion Models via Noisy Query Tokens
- From Illusion to Intention: Visual Rationale Learning for Vision-Language Reasoning
- Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Following
- Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMs
- Revisiting Multimodal KV Cache Compression: A Frequency-Domain-Guided Outlier-KV-Aware Approach
- Mantis: A Versatile Vision-Language-Action Model with Disentangled Visual Foresight
- OpenMMReasoner: Pushing the Frontiers for Multimodal Reasoning with an Open and General Recipe
- OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Models
- VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models
- V-Thinker: Interactive Thinking with Images
- RoboOS-NeXT: A Unified Memory-based Framework for Lifelong, Scalable, and Robust Multi-Robot Collaboration
- ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding
- VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMs
- DynaSolidGeo: A Dynamic Benchmark for Genuine Spatial Mathematical Reasoning of VLMs in Solid Geometry
- Small Drafts, Big Verdict: Information-Intensive Visual Reasoning via Speculation
- Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views
- UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learning
- A Survey on Agentic Multimodal Large Language Models
- ViSurf: Visual Supervised-and-Reinforcement Fine-Tuning for Large Vision-and-Language Models
- One Patch to Caption Them All: A Unified Zero-Shot Captioning Framework
- Bias Beyond Demographics: Probing Decision Boundaries in Black-Box LVLMs via Counterfactual VQA
Discussions
Related