OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM
2025/10/17 by Hanrong Ye, Ye, Hanrong, Chao-Han Huck Yang +61 · 1 voice · 8 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #cs.AI #cs.CL #cs.CV
paper · pdf · doi:10.48550/arxiv.2510.15870
arxiv published 2025/10/17 · arxiv updated 2025/10/27
Abstract
Advancing machine intelligence requires developing the ability to perceive across multiple modalities, much as humans sense the world. We introduce OmniVinci, an initiative to build a strong, open-source, omni-modal LLM. We carefully study the design choices across model architecture and data curation. For model architecture, we present three key innovations: (i) OmniAlignNet for strengthening alignment between vision and audio embeddings in a shared omni-modal latent space; (ii) Temporal Embedding Grouping for capturing relative temporal alignment between vision and audio signals; and (iii) Constrained Rotary Time Embedding for encoding absolute temporal information in omni-modal embeddings. We introduce a curation and synthesis pipeline that generates 24M single-modal and omni-modal conversations. We find that modalities reinforce one another in both perception and reasoning. Our model, OmniVinci, outperforms Qwen2.5-Omni with +19.05 on DailyOmni (cross-modal understanding), +1.7 on MMAR (audio), and +3.9 on Video-MME (vision), while using just 0.2T training tokens - a 6 times reduction compared to Qwen2.5-Omni's 1.2T. We finally demonstrate omni-modal advantages in downstream applications spanning robotics, medical AI, and smart factory.
Citations
- Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models
- Scaling RL to Long Videos
- Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalities
- MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix
- Qwen3 Technical Report
- Multi-Domain Audio Question Answering Toward Acoustic Content Reasoning in The DCASE 2025 Challenge
- Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Models
- InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models
- Video-R1: Reinforcing Video Reasoning in MLLMs
- Qwen2.5-Omni Technical Report
- Scaling Vision Pre-Training to 4K Resolution
- Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities
- Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs
- Investigating and Enhancing Vision-Audio Capability in Omnimodal Large Language Models
- Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction
- Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidance
- Ola: Pushing the Frontiers of Omni-Modal Language Model
- Baichuan-Omni-1.5 Technical Report
- DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
- On the Feasibility of Vision-Language Models for Time-Series Classification
- NVILA: Efficient Frontier Visual Language Models
- NaVILA: Legged Robot Vision-Language-Action Model for Navigation
- NeKo: Cross-Modality Post-Recognition Error Correction with Tasks-Guided Mixture-of-Experts Language Model
- MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark
- OMCAT: Omni Context Aware Transformer
- SPORTU: A Comprehensive Sports Understanding Benchmark for Multimodal Large Language Models
- Low Frame-rate Speech Codec: a Codec Designed for Fast High-quality Speech LLM Training and Inference
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
- LongVILA: Scaling Long-Context Visual Language Models for Long Videos
- VITA: Towards Open-Source Interactive Omni Multimodal LLM
- LLaVA-OneVision: Easy Visual Task Transfer
- Audio Entailment: Assessing Deductive Reasoning for Audio Understanding
- VILA2: VILA Augmented VILA
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding
- Qwen2-Audio Technical Report
- MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captions
- Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation
- Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignment
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
- X-VILA: Cross-Modality Alignment for Large Language Model
- How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites
- When Do We Not Need Larger Vision Models?
- MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
- Gemma: Open Models Based on Gemini Research and Technology
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
- NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation
- BioXP-0.5B: Explainable Medical-AI via RL-GRPO
- Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities
- Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action
- VILA: On Pre-training for Visual Language Models
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark
- WorldSense: A Synthetic Benchmark for Grounded Reasoning in Large Language Models
- Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models
- CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models
- Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
- NExT-GPT: Any-to-Any Multimodal LLM
- A2Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Models
- Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models
- Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding
- AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
- VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset
- Pengi: An Audio Language Model for Audio Tasks
- Any-to-Any Generation via Composable Diffusion
- Listen, Think, and Understand
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
- ImageBind: One Embedding Space To Bind Them All
- mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality
- AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head
- MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
- VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset
- Visual Instruction Tuning
- Sigmoid Loss for Language Image Pre-Training
- PaLM-E: An Embodied Multimodal Language Model
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- Robust Speech Recognition via Large-Scale Weak Supervision
- SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
- Flamingo: a Visual Language Model for Few-Shot Learning
- Clotho-AQA: A Crowdsourced Dataset for Audio Question Answering
- ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
- RoFormer: Enhanced Transformer with Rotary Position Embedding
- Learning Transferable Visual Models From Natural Language Supervision
- VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- FSD50K: An Open Dataset of Human-Labeled Sound Events
- DocVQA: A Dataset for VQA on Document Images
- Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments
- Common Voice: A Massively-Multilingual Speech Corpus
- Clotho: An Audio Captioning Dataset
- The UCR Time Series Archive
- MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations
- A multi-device dataset for urban acoustic scene classification
- Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
- A Diagram Is Worth A Dozen Images
Cited by
Discussions
Related