UniRL: Self-Improving Unified Multimodal Models via Supervised and Reinforcement Learning

2025/05/29 by Weijia Mao, Mao, Weijia, Zhenheng Yang +3 · 8 citations
Computer Science · #Multimodal Machine Learning Applications #Domain Adaptation and Few-Shot Learning #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2505.23380

Abstract

Unified multimodal large language models such as Show-o and Janus have achieved strong performance across both generation and understanding tasks. However, these models typically rely on large-scale datasets and require substantial computation during the pretraining stage. In addition, several post-training methods have been proposed, but they often depend on external data or are limited to task-specific customization. In this work, we introduce UniRL, a self-improving post-training approach. Our approach enables the model to generate images from prompts and use them as training data in each iteration, without relying on any external image data. Moreover, it enables the two tasks to enhance each other: the generated images are used for understanding, and the understanding results are used to supervise generation. We explore supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO) to optimize the models. UniRL offers three key advantages: (1) it requires no external image data, as all training samples are generated by the model itself during training; (2) it not only improves individual task performance, but also reduces the imbalance between generation and understanding; and (3) it requires only several additional training steps during the post-training stage. We evaluate UniRL on top of Show-o and Janus, achieving a GenEval score of 0.77 for Show-o and 0.65 for Janus. Code and models will be released in https://github.com/showlab/UniRL.

Cited by

• Decouple to Generalize: Context-First Self-Evolving Learning for Data-Scarce Vision-Language Reasoning
• Does Understanding Inform Generation in Unified Multimodal Models? From Analysis to Path Forward
• The Image as Its Own Reward: Reinforcement Learning with Adversarial Reward for Image Generation
• AlphaGRPO: Unlocking Self-Reflective Multimodal Generation in UMMs via Decompositional Verifiable Reward
• SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Models
• Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understanding
• Reconstruction Alignment Improves Unified Multimodal Models
• Reinforcement Learning for Large Model: A Survey

Related

• Co-Reinforcement Learning for Unified Multimodal Understanding and Generation
• SUDER: Self-Improving Unified Large Multimodal Models for Understanding and Generation with Dual Self-Rewards
• UniRL-Zero: Reinforcement Learning on Unified Models with Joint Language Model and Diffusion Model Experts
• Janus-Pro-R1: Advancing Collaborative Visual Comprehension and Generation via Reinforcement Learning
• SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Models