PixArt-Σ: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation
2024/03/07 by Jun Song Chen, Junsong Chen, Chen, Junsong +18 · 2 voices · 135 citations
Computer Science · #Advanced Data Compression Techniques #Artificial intelligence #Computer science #Economics #Image (mathematics) #Operations management #Physics #Quantum mechanics #Sigma #Six Sigma #cs.CV
paper · pdf · doi:10.48550/arxiv.2403.04692
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2024/03/07 · openalex created_date 2024/03/09 · openalex updated_date 2026/07/28
Abstract
In this paper, we introduce PixArt-Σ, a Diffusion Transformer model~(DiT) capable of directly generating images at 4K resolution. PixArt-Σrepresents a significant advancement over its predecessor, PixArt-α, offering images of markedly higher fidelity and improved alignment with text prompts. A key feature of PixArt-Σis its training efficiency. Leveraging the foundational pre-training of PixArt-α, it evolves from the `weaker' baseline to a `stronger' model via incorporating higher quality data, a process we term "weak-to-strong training". The advancements in PixArt-Σare twofold: (1) High-Quality Training Data: PixArt-Σincorporates superior-quality image data, paired with more precise and detailed image captions. (2) Efficient Token Compression: we propose a novel attention module within the DiT framework that compresses both keys and values, significantly improving efficiency and facilitating ultra-high-resolution image generation. Thanks to these improvements, PixArt-Σachieves superior image quality and user prompt adherence capabilities with significantly smaller model size (0.6B parameters) than existing text-to-image diffusion models, such as SDXL (2.6B parameters) and SD Cascade (5.1B parameters). Moreover, PixArt-Σ's capability to generate 4K images supports the creation of high-resolution posters and wallpapers, efficiently bolstering the production of high-quality visual content in industries such as film and gaming.
Cited by
- Marrying Autoregressive Transformer and Diffusion with Multi-Reference Autoregression
- Importance-Aware OBS Pruning for Diffusion Models
- Text Template Tokens Are Implicit Semantic Registers in Diffusion Transformers
- Uni-AdaVD: Universal Concept Erasure for Visual Generation via Orthogonal Value Decomposition
- Stable Audio 3
- MIRO: MultI-Reward cOnditioned pretraining improves T2I quality and efficiency
- How far can we go with ImageNet for Text-to-Image generation?
- LongCat-Image Technical Report
- ThinkGen: Generalized Thinking for Visual Generation
- Boosting Monocular Metric Depth Estimation via Bokeh Rendering
- TriDF: Evaluating Perception, Detection, and Hallucination for Interpretable DeepFake Detection
- UltraImage: Rethinking Resolution Extrapolation in Image Diffusion Transformers
- Zero-Shot Video Translation and Editing with Frame Spatial-Temporal Correspondence
- TwinFlow: Realizing One-step Generation on Large Models with Self-adversarial Flows
- Glance: Accelerating Diffusion Models with 1 Sample
- VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstruction
- GOATex: Geometry & Occlusion-Aware Texturing
- Visual Puns from Idioms: An Iterative LLM-T2IM-MLLM Framework
- PixelDiT: Pixel Diffusion Transformers for Image Generation
- Beyond Words and Pixels: A Benchmark for Implicit World Knowledge Reasoning in Generative Models
- UltraFlux: Data-Model Co-Design for High-quality Native 4K Text-to-Image Generation across Diverse Aspect Ratios
- TRIM: Scalable 3D Gaussian Diffusion Inference with Temporal and Spatial Trimming
- SplitFlux: Learning to Decouple Content and Style from a Single Image
- FreeSwim: Revisiting Sliding-Window Attention Mechanisms for Training-Free Ultra-High-Resolution Video Generation
- Mixture of States: Routing Token-Level Dynamics for Multimodal Generation
- ImAgent: A Unified Multimodal Agent Framework for Test-Time Scalable Image Generation
- Generative AI Meets 6G and Beyond: Diffusion Models for Semantic Communications
- STaMP: Sequence Transformation and Mixed Precision for Low-Precision Activation Quantization
- ScaleDiff: Higher-Resolution Image Synthesis via Efficient and Model-Agnostic Diffusion
- Product-Quantised Image Representation for High-Quality Image Synthesis
- AlphaGRPO: Unlocking Self-Reflective Multimodal Generation in UMMs via Decompositional Verifiable Reward
- UltraImageGen: Efficient Ultra-High-Resolution Image Generation with Hierarchical Local Attention
- More Than Generation: Unifying Generation and Depth Estimation via Text-to-Image Diffusion Models
- UniAIDet: A Unified and Universal Benchmark for AI-Generated Image Content Detection and Localization
- M3T2IBench: A Large-Scale Multi-Category, Multi-Instance, Multi-Relation Text-to-Image Benchmark
- UltraHR-100K: Enhancing UHR Image Synthesis with A Large-Scale High-Quality Dataset
- D2D: Detector-to-Differentiable Critic for Improved Numeracy in Text-to-Image Generation
- UniRL-Zero: Reinforcement Learning on Unified Models with Joint Language Model and Diffusion Model Experts
- L2P: Unlocking Latent Potential for Pixel Generation
- FraQAT: Quantization Aware Training with Fractional bits
- MetaCaptioner: Towards Generalist Visual Captioning with Open-source Suites
- Asymmetric Flow Models
- Mitigating Diffusion Model Hallucinations with Dynamic Guidance
- Let Features Decide Their Own Solvers: Hybrid Feature Caching for Diffusion Transformers
- Query-Kontext: An Unified Multimodal Model for Image Generation and Editing
- Post-Training Quantization via Residual Truncation and Zero Suppression for Diffusion Models
- CO3: Contrasting Concepts Compose Better
- DC-Gen: Post-Training Diffusion Acceleration with Deeply Compressed Latent Space
- SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformer
- Un-Doubling Diffusion: LLM-guided Disambiguation of Homonym Duplication
- OmniBridge: Unified Multimodal Understanding, Generation, and Retrieval via Latent Space Alignment
- Audio Super-Resolution with Latent Bridge Models
- MaskAttn-SDXL: Controllable Region-Level Text-To-Image Generation
- Moment- and Power-Spectrum-Based Gaussianity Regularization for Text-to-Image Models
- OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generation
- Easier Painting Than Thinking: Can Text-to-Image Models Set the Stage, but Not Direct the Play?
- InfoScale: Unleashing Training-free Variable-scaled Image Generation via Effective Utilization of Information
- Leveraging Diffusion Models for Stylization using Multiple Style Images
- Navigating the Exploration-Exploitation Tradeoff in Inference-Time Scaling of Diffusion Models
- Error Propagation Mechanisms and Compensation Strategies for Quantized Diffusion
- Match & Choose: Model Selection Framework for Fine-tuning Text-to-Image Diffusion Models
- TBAC-UniImage: Unified Understanding and Generation by Ladder-Side Diffusion Tuning
- Fewer Denoising Steps or Cheaper Per-Step Inference: Towards Compute-Optimal Diffusion Model Deployment
- Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing
- HPSv3: Towards Wide-Spectrum Human Preference Score
- LRQ-DiT: Log-Rotation Post-Training Quantization of Diffusion Transformers for Image and Video Generation
- Qwen-Image Technical Report
- UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editing
- Trade-offs in Image Generation: How Do Different Dimensions Interact?
- FreeCus: Free Lunch Subject-driven Customization in Diffusion Transformers
- Identifying Prompted Artist Names from Generated Images
- Think in Strokes, Not Pixels: Process-Driven Image Generation via Interleaved Reasoning
- Sparse Fine-Tuning of Transformers for Generative Tasks
- Inversion-DPO: Precise and Efficient Post-Training for Diffusion Models
- Upsample What Matters: Region-Adaptive Latent Sampling for Accelerated Diffusion Transformers
- SV-DRR: High-Fidelity Novel View X-Ray Synthesis Using Diffusion Model
- DC-AR: Efficient Masked Autoregressive Image Generation with Deep Compression Hybrid Tokenizer
- Structured Captions Improve Prompt Adherence in Text-to-Image Models (Re-LAION-Caption 19M)
- AIGI-Holmes: Towards Explainable and Generalizable AI-Generated Image Detection via Multimodal Large Language Models
- Prompt Reinjection: Alleviating Prompt Forgetting in Multimodal Diffusion Transformers for Text-to-Image Generation
- Instella-T2I: Pushing the Limits of 1D Discrete Latent Space Image Generation
- Distillation-Enabled Knowledge Alignment for Generative Semantic Communications in AIGC Provisioning Tasks
- VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Models
- UniFork: Exploring Modality Alignment for Unified Multimodal Understanding and Generation
- The Hidden Cost of an Image: Quantifying the Energy Consumption of AI Image Generation
- Nabla-R2D3: Effective and Efficient 3D Diffusion Alignment with 2D Rewards
- DDiT: Dynamic Resource Allocation for Diffusion Transformer Model Serving
- Interpretable and Reliable Detection of AI-Generated Images via Grounded Reasoning in MLLMs
- Where and How to Perturb: On the Design of Perturbation Guidance in Diffusion and Flow Models
- SPARKE: Scalable Prompt-Aware Diversity and Novelty Guidance in Diffusion Models via RKE Score
- HadaNorm: Diffusion Transformer Quantization through Mean-Centered Transformations
- PixCell: A generative foundation model for digital histopathology images
- RAID: A Dataset for Testing the Adversarial Robustness of AI-Generated Image Detectors
- Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences
- Ultra-High-Resolution Image Synthesis: Data, Method and Evaluation
- TIIF-Bench: How Does Your T2I Model Follow Your Instructions?
- Latent Wavelet Diffusion For Ultra-High-Resolution Image Synthesis
- Multi-Group Proportional Representation for Text-to-Image Models
- Re-ttention: Ultra Sparse Visual Generation via Attention Statistical Reshape
- PacTure: Efficient PBR Texture Generation on Packed Views with Visual Autoregressive Models
- ISAC: Training-Free Instance-to-Semantic Attention Control for Multi-Instance Generation
- Score Replacement with Bounded Deviation for Rare Prompt Generation
- RainFusion: Adaptive Video Generation Acceleration via Multi-Dimensional Visual Redundancy
- Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities
- Multimodal LLM-Guided Semantic Correction in Text-to-Image Diffusion
- MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Models
- FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocities
- Jodi: Unification of Visual Generation and Understanding via Joint Modeling
- Shifting AI Efficiency From Model-Centric to Data-Centric Compression
- Model Already Knows the Best Noise: Bayesian Active Noise Selection via Attention in Video Diffusion Model
- RePrompt: Reasoning-Augmented Reprompting for Text-to-Image Generation via Reinforcement Learning
- From Prediction to Perfection: Introducing Refinement to Autoregressive Image Generation
- T2I-ConBench: Text-to-Image Benchmark for Continual Post-training
- Panoptic Captioning: An Equivalence Bridge for Image and Text
- Scaling Diffusion Transformers Efficiently via μP
- Grouping First, Attending Smartly: Training-Free Acceleration for Diffusion Transformers
- SounDiT: Geo-Contextual Soundscape-to-Landscape Generation
- UDT: Reconciling U-Nets and Diffusion Transformers with Data-Adaptive Token Reduction
- DRAGON: A Large-Scale Dataset of Realistic Images Generated by Diffusion Models
- CheXGenBench: A Unified Benchmark For Fidelity, Privacy and Utility of Synthetic Chest Radiographs
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis
- Improving Editability in Image Generation with Layer-wise Memory
- BitDance: Scaling Autoregressive Generative Models with Binary Tokens
- Can We Achieve Efficient Diffusion without Self-Attention? Distilling Self-Attention into Convolutions
- AGHI-QA: A Subjective-Aligned Dataset and Metric for AI-Generated Human Images
- PiD: Fast and High-Resolution Latent Decoding with Pixel Diffusion
- Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation
- Where Do the Joules Go? Diagnosing Inference Energy Consumption
- Text-to-Image Alignment in Denoising-Based Models through Step Selection
- From Reflection to Perfection: Scaling Inference-Time Optimization for Text-to-Image Diffusion Models via Reflection Tuning
- Acquire and then Adapt: Squeezing out Text-to-Image Model for Image Restoration
- Turbo2K: Towards Ultra-Efficient and High-Quality 2K Video Synthesis
- LMM4LMM: Benchmarking and Evaluating Large-multimodal Image Generation with LMMs
- MixDiT: Accelerating Image Diffusion Transformer Inference with Mixed-Precision MX Quantization
- HiFlow: Training-free High-Resolution Image Generation with Flow-Aligned Guidance
Discussions
Related