Resurrect Mask AutoRegressive Modeling for Efficient and Scalable Image Generation

2025/07/17 by Xin Yi, Le Zhuo, Xin, Yi +19 · 8 citations
Computer Science · Engineering · #Advanced Image and Video Retrieval Techniques #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Industrial Vision Systems and Defect Detection

paper · pdf · doi:10.48550/arxiv.2507.13032

openalex publication_date 2025/07/17 · openalex created_date 2025/10/14 · openalex updated_date 2026/07/28

Abstract

AutoRegressive (AR) models have made notable progress in image generation, with Masked AutoRegressive (MAR) models gaining attention for their efficient parallel decoding. However, MAR models have traditionally underperformed when compared to standard AR models. This study refines the MAR architecture to improve image generation quality. We begin by evaluating various image tokenizers to identify the most effective one. Subsequently, we introduce an improved Bidirectional LLaMA architecture by replacing causal attention with bidirectional attention and incorporating 2D RoPE, which together form our advanced model, MaskGIL. Scaled from 111M to 1.4B parameters, MaskGIL achieves a FID score of 3.71, matching state-of-the-art AR models in the ImageNet 256x256 benchmark, while requiring only 8 inference steps compared to the 256 steps of AR models. Furthermore, we develop a text-driven MaskGIL model with 775M parameters for generating images from text at various resolutions. Beyond image generation, MaskGIL extends to accelerate AR-based generation and enable real-time speech-to-image conversion. Our codes and models are available at https://github.com/synbol/MaskGIL.

Citations

• From Reflection to Perfection: Scaling Inference-Time Optimization for Text-to-Image Diffusion Models via Reflection Tuning
• Lumina-Image 2.0: A Unified and Efficient Image Generative Framework
• Customize Your Visual Autoregressive Recipe with Set Autoregressive Modeling
• Open-MAGVIT2: An Open-Source Project Toward Democratizing Auto-regressive Visual Generation
• Scalable Autoregressive Image Generation with Mamba
• Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining
• Gemma 2: Improving Open Language Models at a Practical Size
• Autoregressive Image Generation without Vector Quantization
• Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation
• Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiT
• Chameleon: Mixed-Modal Early-Fusion Foundation Models
• Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers
• Video-LLaVA: Learning United Visual Representation by Alignment Before Projection
• A Pytorch Reproduction of Masked Generative Image Transformer
• GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment
• Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation
• PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis
• Stay on topic with Classifier-Free Guidance
• Visual Instruction Tuning
• GPT-4 Technical Report
• StraIT: Non-autoregressive Generation with Stratified Image Transformer
• LLaMA: Open and Efficient Foundation Language Models
• Scaling Vision Transformers to 22 Billion Parameters
• Muse: Text-To-Image Generation via Masked Generative Transformers
• Scalable Diffusion Models with Transformers
• Robust Speech Recognition via Large-Scale Weak Supervision
• MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis
• Improved Masked Image Generation with Token-Critic
• Classifier-Free Diffusion Guidance
• Scaling Autoregressive Models for Content-Rich Text-to-Image Generation
• Autoregressive Image Generation using Residual Quantization
• MaskGIT: Masked Generative Image Transformer
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• CogView: Mastering Text-to-Image Generation via Transformers
• Diffusion Models Beat GANs on Image Synthesis
• Learning Transferable Visual Models From Natural Language Supervision
• Zero-Shot Text-to-Image Generation
• Taming Transformers for High-Resolution Image Synthesis
• Language Models are Few-Shot Learners
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• Improved Precision and Recall Metric for Assessing Generative Models
• Neural Discrete Representation Learning
• Attention Is All You Need
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• dMLLM-TTS: Self-Verified and Efficient Test-Time Scaling for Diffusion Multi-Modal Large Language Models
• Multi-Agent Medical Decision Consensus Matrix System: An Intelligent Collaborative Framework for Oncology MDT Consultations
• HUD: Hierarchical Uncertainty-Aware Disambiguation Network for Composed Video Retrieval
• Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understanding
• Multi-Modal Machine Learning Framework for Predicting Early Recurrence of Brain Tumors Using MRI and Clinical Biomarkers
• A Multimodal Deep Learning Framework for Early Diagnosis of Liver Cancer via Optimized BiLSTM-AM-VMD Architecture
• Low-Cost Test-Time Adaptation for Robust Video Editing
• Lumina-mGPT 2.0: Stand-Alone AutoRegressive Image Modeling
• Neural-Driven Image Editing
• ReDDiT: Rehashing Noise for Discrete Visual Generation
• LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Model

Related

• DC-AR: Efficient Masked Autoregressive Image Generation with Deep Compression Hybrid Tokenizer
• MARché: Fast Masked Autoregressive Image Generation with Cache-Aware Attention
• LazyMAR: Accelerating Masked Autoregressive Models via Feature Caching
• Token-Shuffle: Towards High-Resolution Image Generation with Autoregressive Models
• Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation