UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddings

2025/11/01 by Zhibin Lan, Liqiang Niu, Lan, Zhibin +6 · 3 citations
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2511.00405

openalex publication_date 2025/11/01 · openalex created_date 2025/11/05 · openalex updated_date 2026/07/28

Abstract

The remarkable success of multimodal large language models (MLLMs) has driven advances in multimodal embeddings, yet existing models remain inherently discriminative, limiting their ability to benefit from reasoning-driven generation paradigm. In this work, we pioneer the exploration of generative embeddings, unifying embedding tasks within a generative paradigm. We propose UME-R1, a universal multimodal embedding framework consisting of a two-stage training strategy: a cold-start supervised fine-tuning equips the model with reasoning capabilities and enables it to generate both discriminative and generative embeddings; a subsequent reinforcement learning enhances reasoning and further optimizes generative embedding quality. This pioneering work reveals four key insights: 1) generative embeddings unlock substantial performance gains over conventional discriminative embeddings by leveraging the powerful generative reasoning capabilities of MLLMs; 2) discriminative and generative embeddings are complementary, whose combined oracle performance far exceeding that of either alone; 3) RL can effectively enhance generative embeddings, establishing a scalable optimization paradigm.; 4) repeated sampling at inference boosts downstream task coverage (pass@k), highlighting the inference-time scalability potential of generative embeddings. Evaluated on the MMEB-V2 benchmark across 78 tasks spanning video, image, and visual documents, UME-R1 significantly outperforms conventional discriminative embedding models and offers a foundation for more interpretable, reasoning-driven generative multimodal embeddings. Our code, models, and datasets will be publicly available at https://github.com/XMUDeepLIT/UME-R1.

Citations

• From Generator to Embedder: Harnessing Innate Abilities of Multimodal LLMs via Building Zero-Shot Discriminative Embedding Model
• VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents
• Improve Multi-Modal Embedding Learning via Explicit Hard Negative Gradient Amplifying
• Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Mining
• Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs
• VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model
• CAFe: Unifying Representation and Generation with Contrastive-Autoregressive Finetuning
• Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RL
• R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Model
• LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learning
• Visual-RFT: Visual Reinforcement Fine-Tuning
• Investigating Inference-time Scaling for Chain of Multi-modal Thought: A Preliminary Study
• mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Data
• s1: Simple test-time scaling
• MegaPairs: Massive Data Synthesis For Universal Multimodal Retrieval
• VladVA: Discriminative Fine-tuning of LVLMs
• LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant
• MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs
• GPT-4o System Card
• VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents
• VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks
• AVG-LLaVA: An Efficient Large Multimodal Model with Adaptive Visual Granularity
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• LLaVA-OneVision: Easy Visual Task Transfer
• Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
• E5-V: Universal Embeddings with Multimodal Large Language Models
• ColPali: Efficient Document Retrieval with Vision Language Models
• Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward
• MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
• UniIR: Training and Benchmarking Universal Multimodal Information Retrievers
• Visual Instruction Tuning
• Sigmoid Loss for Language Image Pre-Training
• Retrieving Multimodal Information for Augmented Generation: A Survey
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• When and why vision-language models behave like bags-of-words, and what to do about it?
• ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models
• Learning Transferable Visual Models From Natural Language Supervision
• OK-VQA: A Visual Question Answering Benchmark Requiring External\n Knowledge
• Visual7W: Grounded Question Answering in Images
• GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning

Cited by

• ReLoop-UME: Recurrent Depth with Learnable Retrieval Registers for Universal Multimodal Embedding
• Douyin Multimodal Embedding Model Technical Report
• MVEB: Massive Video Embedding Benchmark
• Learning from Failures: Retrieval-Centric CoT via Hard Negatives for Unified Multimodal Retrieval

Related

• Think Then Embed: Generative Context Improves Multimodal Embedding
• Reasoning Guided Embeddings: Leveraging MLLM Reasoning for Improved Multimodal Retrieval
• Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings
• Generative Multimodal Models are In-Context Learners
• From Generator to Embedder: Harnessing Innate Abilities of Multimodal LLMs via Building Zero-Shot Discriminative Embedding Model