A High-Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generation

2025/06/11 by Yulong Feng, Feng, Yukang, Jianwen Sun +19
Computer Science · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2506.09427

openalex publication_date 2025/06/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent advancements in Large Multimodal Models (LMMs) have significantly improved multimodal understanding and generation. However, these models still struggle to generate tightly interleaved image-text outputs, primarily due to the limited scale, quality, and instructional richness of current training datasets. To address this, we introduce InterSyn, a dataset that features: (1) large scale, comprising 1.8M multimodal samples; (2) high quality, supported by our proposed Self-Evaluation with Iterative Refinement (SEIR) method for rigorous automated quality refinement; (3) rich instructional diversity, ensured through diverse well-designed question templates, based on human preferences and covering a 3500-topic hierarchy. These characteristics make InterSyn particularly well-suited for training LMMs in interactive image-text generation capabilities. To evaluate the capabilities, we propose SynJudge, a reliable automatic evaluator that aligns closely with human judge and outputs four interpretable scores: Text Content Completeness (TCC), Image Content Completeness (ICC), Image Quality (IQ), and Image-Text Synergy (ITS). These scores are complementary, covering both content and quality as well as cross-modal interaction, thereby forming a comprehensive evaluation framework. Experimental results on InterSyn subsets of up to 200K samples show that 25K-50K already yield substantial improvements, while scaling to 100K/200K brings further gains in TCC, ICC, and especially ITS, highlighting InterSyn's: (1) scalability, as performance consistently improves with more data; (2) efficiency, as significant gains are achievable even with smaller subsets, making it accessible to researchers with varying computational resources.

Citations

• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• Emerging Properties in Unified Multimodal Pretraining
• VARGPT-v1.1: Improve Visual Autoregressive Large Unified Model via Iterative Instruction Tuning and Reinforcement Learning
• Qwen2.5-VL Technical Report
• Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling
• VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Model
• Dual Diffusion for Unified Image Generation and Understanding
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
• Liquid: Language Models are Scalable and Unified Multi-modal Generators
• Interleaved Scene Graphs for Interleaved Text-and-Image Generation Assessment
• MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models
• Emu3: Next-Token Prediction is All You Need
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation
• Show-o: One Single Transformer to Unify Multimodal Understanding and Generation
• SEED-Story: Multimodal Long Story Generation with Large Language Model
• ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generation
• Holistic Evaluation for Interleaved Text-and-Image Generation
• CoMM: A Coherent Interleaved Image-Text Dataset for Multimodal Understanding and Generation
• Chameleon: Mixed-Modal Early-Fusion Foundation Models
• Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
• InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
• MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
• ShareGPT4V: Improving Large Multi-Modal Models with Better Captions
• MMBench: Is Your Multi-modal Model an All-around Player?
• MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
• OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents
• Revisiting the Role of Language Priors in Vision-Language Models
• Intelligent Grimm -- Open-ended Visual Storytelling via Latent Diffusion Models
• X-IQE: eXplainable Image Quality Evaluation for Text-to-Image Generation with Visual Large Language Models
• LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluation
• Visual Instruction Tuning
• Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Text
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• Flamingo: a Visual Language Model for Few-Shot Learning
• Hierarchical Text-Conditional Image Generation with CLIP Latents
• Visual Goal-Step Inference using wikiHow
• Towards Automatic Learning of Procedures from Web Instructional Videos
• Improved Techniques for Training GANs
• Visual Storytelling
• Modality-Specialized Synergizers for Interleaved Vision-Language Generalists
• OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation
• MiniGPT-5: Interleaved Vision-and-Language Generation via Generative Vokens

Related

• OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation
• CoMM: A Coherent Interleaved Image-Text Dataset for Multimodal Understanding and Generation
• Holistic Evaluation for Interleaved Text-and-Image Generation
• Interleaved Scene Graphs for Interleaved Text-and-Image Generation Assessment
• ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generation