Jointly Training Large Autoregressive Multimodal Models

2023/09/27 by Emanuele Aiello, Aiello, Emanuele, Lili Yu +7 · 2 citations
Computer Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2309.15564

openalex publication_date 2023/09/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In recent years, advances in the large-scale pretraining of language and text-to-image models have revolutionized the field of machine learning. Yet, integrating these two modalities into a single, robust model capable of generating seamless multimodal outputs remains a significant challenge. To address this gap, we present the Joint Autoregressive Mixture (JAM) framework, a modular approach that systematically fuses existing text and image generation models. We also introduce a specialized, data-efficient instruction-tuning strategy, tailored for mixed-modal generation tasks. Our final instruct-tuned model demonstrates unparalleled performance in generating high-quality multimodal outputs and represents the first model explicitly designed for this purpose.

Cited by

• Harmonizing and Merging Source Models for CLIP-based Domain Generalization
• MMaDA: Multimodal Large Diffusion Language Models

Related

• Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning
• STAR: STacked AutoRegressive Scheme for Unified Multimodal Learning
• ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generation
• ARMOR: Empowering Multimodal Understanding Model with Interleaved Multimodal Generation Capability
• MENTOR: Efficient Multimodal-Conditioned Tuning for Autoregressive Vision Generation Models