Distilling Specialized Orders for Visual Generation

2025/04/23 by Rishav Pramanik, Pramanik, Rishav, Sghaier, Amin +8 · 1 citation
Computer Science · #Advanced Image and Video Retrieval Techniques #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Image Retrieval and Classification Techniques

paper · pdf · doi:10.48550/arxiv.2504.17069

openalex publication_date 2025/04/23 · openalex created_date 2025/10/18 · openalex updated_date 2026/07/28

Abstract

Autoregressive (AR) image generators are becoming increasingly popular due to their ability to produce high-quality images and their scalability. Typical AR models are locked onto a specific generation order, often a raster-scan from top-left to bottom-right; this prohibits multi-task flexibility (inpainting, editing, outpainting) without retraining. Any-order AR models address this by learning to generate under arbitrary patch orderings, but at the cost of increased complexity and lower performance. In this paper, we present Ordered Autoregressive (OAR) generation, a self-distillation pipeline that first trains an any-order AR model, then extracts specialized generation orders from the model's own confidence scores, and fine-tunes on these orders. This achieves two goals: 1) improved generation quality by redirecting capacity from learning all N! orderings to a single specialized path, and 2) preserved flexibility of any-order models. On ImageNet 256× 256, OAR improves FID from 2.39 to 2.17 over the any-order baseline, with consistent gains on Fashion Products and CelebA-HQ. OAR supports zero-shot inpainting and outpainting without retraining, and human evaluation shows 64% preference over the baseline. The pipeline requires only lightweight fine-tuning on a pretrained any-order model, with no architectural changes or additional annotations.

Citations

• Qwen3 Technical Report
• Learning-Order Autoregressive Models with Application to Molecular Graph Generation
• RandAR: Decoder-only Autoregressive Visual Generation in Random Orders
• Randomized Autoregressive Visual Generation
• Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens
• Customize Your Visual Autoregressive Recipe with Set Autoregressive Modeling
• Autoregressive Image Generation without Vector Quantization
• An Image is Worth 32 Tokens for Reconstruction and Generation
• Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation
• σ-GPTs: A New Approach to Autoregressive Models
• Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction
• Arrows of Time for Large Language Models
• SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers
• Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation
• Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
• Scaling up GANs for Text-to-Image Synthesis
• Scalable Diffusion Models with Transformers
• MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis
• Efficiently Scaling Transformer Inference
• Training and Inference on Any-Order Autoregressive Models the Right Way
• The Role of ImageNet Classes in Fréchet Inception Distance
• MaskGIT: Masked Generative Image Transformer
• StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• Discovering Non-monotonic Autoregressive Orderings with Variational Inference
• Autoregressive Diffusion Models
• A Review of the Gumbel-max Trick and its Extensions for Discrete Stochasticity in Machine Learning
• Structured Denoising Diffusion Models in Discrete State-Spaces
• Diffusion Models Beat GANs on Image Synthesis
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Zero-Shot Text-to-Image Generation
• Position Information in Transformers: An Overview
• Position Information in Transformers: An Overview
• Taming Transformers for High-Resolution Image Synthesis
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Language Models are Few-Shot Learners
• A Theory of Usable Information Under Computational Constraints
• Axial Attention in Multidimensional Transformers
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• Stochastic Beams and Where to Find Them: The Gumbel-Top-k Trick for Sampling Sequences Without Replacement
• Non-Monotonic Sequential Text Generation
• Insertion-based Decoding with automatically Inferred Generation Order
• Middle-Out Decoding
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• The Importance of Generation Order in Language Modeling
• A Tree-based Decoder for Neural Machine Translation
• Self-Attention with Relative Position Representations
• Image Transformer
• PixelSNAIL: An Improved Autoregressive Generative Model
• Neural Discrete Representation Learning
• Attention Is All You Need
• Bidirectional Beam Search: Forward-Backward Inference in Neural Sequence Models for Fill-in-the-Blank Image Captioning
• Layer Normalization
• Conditional Image Generation with PixelCNN Decoders
• Neural Autoregressive Distribution Estimation
• Pixel Recurrent Neural Networks
• Order Matters: Sequence to sequence for sets
• MADE: Masked Autoencoder for Distribution Estimation
• A Deep and Tractable Density Estimator
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• Randomized Autoregressive Visual Generation
• Next Patch Prediction for Autoregressive Visual Generation
• Autoregressive Image Generation with Vision Full-view Prompt
• AR-RAG: Autoregressive Retrieval Augmentation for Image Generation
• Autoregressive Image Generation with Linear Complexity: A Spatial-Aware Decay Perspective