Rethinking Generative Image Pretraining: How Far Are We From Scaling Up Next-Pixel Prediction?

2025/11/11 by Xinchen Yan, Yan, Xinchen, Chen Liang +10 · 1 voice
Computer Science · Neuroscience · #Generative Adversarial Networks and Image Synthesis #Image and Video Quality Assessment #Visual perception and processing mechanisms #cs.CV #cs.LG

paper · pdf · doi:10.48550/arxiv.2511.08704

arxiv published 2025/11/11 · arxiv updated 2026/05/17

Abstract

This paper investigates the scaling properties of autoregressive next-pixel prediction, a simple, end-to-end yet under-explored framework for unified vision models. Starting with images at resolutions of 32x32, we train a family of Transformers using IsoFlops profiles across compute budgets up to 7e19 FLOPs and evaluate three distinct target metrics: next-pixel prediction objective, ImageNet classification accuracy, and generation-based completion measured by Fr'echet Distance. First, optimal scaling strategy is critically task-dependent. At a fixed resolution of 32x32 alone, the optimal scaling properties for image classification and image generation diverge, where generation optimal setup requires the data size grow three to five times faster than for the classification optimal setup. Second, as image resolution increases, the optimal scaling strategy indicates that the model size must grow much faster than data size. Surprisingly, by projecting our findings, we discover that the primary bottleneck is compute rather than the amount of training data. As compute continues to grow four to five times annually, we forecast the feasibility of pixel-by-pixel modeling of images within the next five years.

Citations

• Neighboring Autoregressive Modeling for Efficient Visual Generation
• Fractal Generative Models
• Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More
• Language-Guided Image Tokenization for Generation
• HART: Efficient Visual Generation with Hybrid Autoregressive Transformer
• The Llama 3 Herd of Models
• An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels
• An Image is Worth 32 Tokens for Reconstruction and Generation
• Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation
• Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction
• Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass Diffusion Transformers
• VideoPoet: A Large Language Model for Zero-Shot Video Generation
• Rejuvenating image-GPT as Strong Visual Representation Learners
• PaLM 2 Technical Report
• DINOv2: Learning Robust Visual Features without Supervision
• Scaling Vision Transformers to 22 Billion Parameters
• Simple diffusion: End-to-end diffusion for high resolution images
• MAGVIT: Masked Generative Video Transformer
• Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
• Training Compute-Optimal Large Language Models
• MaskGIT: Masked Generative Image Transformer
• High-Resolution Image Synthesis with Latent Diffusion Models
• Masked Autoencoders Are Scalable Vision Learners
• Vector-quantized Image Modeling with Improved VQGAN
• BEiT: BERT Pre-Training of Image Transformers
• Scaling Vision Transformers
• Cascaded Diffusion Models for High Fidelity Image Generation
• Diffusion Models Beat GANs on Image Synthesis
• Emerging Properties in Self-Supervised Vision Transformers
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Improved Denoising Diffusion Probabilistic Models
• Taming Transformers for High-Resolution Image Synthesis
• Score-Based Generative Modeling through Stochastic Differential Equations
• Scaling Laws for Autoregressive Generative Modeling
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Denoising Diffusion Implicit Models
• Denoising Diffusion Probabilistic Models
• GLU Variants Improve Transformer
• Scaling Laws for Neural Language Models
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• Neural Discrete Representation Learning
• Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
• Attention Is All You Need
• PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications
• Xception: Deep Learning with Depthwise Separable Convolutions
• Conditional Image Generation with PixelCNN Decoders
• Pixel Recurrent Neural Networks
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• Distilling the Knowledge in a Neural Network
• Going Deeper with Convolutions
• Sequence to Sequence Learning with Neural Networks

Discussions

• How far are we from scaling up next-pixel prediction? [hn, 1 points, 0 comments]

Related

• Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction
• Rethinking Training Dynamics in Scale-wise Autoregressive Generation
• Scalable Pre-training of Large Autoregressive Image Models
• Scaling Backwards: Minimal Synthetic Pre-training?
• Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?