Conditional Image Generation with PixelCNN Decoders

2016/06/16 by Aäron van den Oord, Oord, Aaron van den, Nal Kalchbrenner +9 · 155 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Human Pose and Action Recognition #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.1606.05328

openalex publication_date 2016/06/16 · openalex created_date 2019/07/30 · openalex updated_date 2026/07/28

Abstract

This work explores conditional image generation with a new image density model based on the PixelCNN architecture. The model can be conditioned on any vector, including descriptive labels or tags, or latent embeddings created by other networks. When conditioned on class labels from the ImageNet database, the model is able to generate diverse, realistic scenes representing distinct animals, objects, landscapes and structures. When conditioned on an embedding produced by a convolutional network given a single image of an unseen face, it generates a variety of new portraits of the same person with different facial expressions, poses and lighting conditions. We also show that conditional PixelCNN can serve as a powerful decoder in an image autoencoder. Additionally, the gated convolutional layers in the proposed model improve the log-likelihood of PixelCNN to match the state-of-the-art performance of PixelRNN on ImageNet, with greatly reduced computational cost.

Citations

• Neural GPUs Learn Algorithms

Cited by

• A Neural Network-Based Real-time Casing Collar Recognition System for Downhole Instruments
• Visual Autoregressive Modelling for Monocular Depth Estimation
• UMAMI: Unifying Masked Autoregressive Models and Deterministic Rendering for View Synthesis
• VA-π: Variational Policy Alignment for Pixel-Aware Autoregressive Generation
• Spherical Leech Quantization for Visual Tokenization and Generation
• Robustness of Probabilistic Models to Low-Quality Data: A Multi-Perspective Analysis
• Fast-ARDiff: An Entropy-informed Acceleration Framework for Continuous Space Autoregressive Generation
• SJD++: Improved Speculative Jacobi Decoding for Training-free Acceleration of Discrete Auto-regressive Text-to-Image Generation
• Self-Supervised AI-Generated Image Detection: A Camera Metadata Perspective
• Markovian Scale Prediction: A New Era of Visual Autoregressive Generation
• The Collapse of Patches
• DiverseVAR: Balancing Diversity and Quality of Next-Scale Visual Autoregressive Models
• A Compact Hybrid Convolution--Frequency State Space Network for Learned Image Compression
• AMS-KV: Adaptive KV Caching in Multi-Scale Visual Autoregressive Transformers
• Masked Auto-Regressive Variational Acceleration: Fast Inference Makes Practical Reinforcement Learning
• ActVAR: Activating Mixtures of Weights and Tokens for Efficient Visual Autoregressive Generation
• Rethinking Generative Image Pretraining: How Far Are We From Scaling Up Next-Pixel Prediction?
• Hawk: Leveraging Spatial Context for Faster Autoregressive Text-to-Image Generation
• Multi-scale Autoregressive Models are Laplacian, Discrete, and Latent Diffusion Models in Disguise
• FARMER: Flow AutoRegressive Transformer over Pixels
• JiuTian Chuanliu: A Large Spatiotemporal Model for General-purpose Dynamic Urban Sensing
• Scaling Non-Parametric Sampling with Representation
• Self-diffusion for Solving Inverse Problems
• Distilled Decoding 2: One-step Sampling of Image Auto-regressive Models with Conditional Score Distillation
• EditInfinity: Image Editing with Binary-Quantized Generative Models
• Video Consistency Distance: Enhancing Temporal Consistency for Image-to-Video Generation via Reward-Based Fine-Tuning
• Generation then Reconstruction: Accelerating Masked Autoregressive Models via Two-Stage Sampling
• PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows
• Spoof Face Detection Via Semi-Supervised Adversarial Training
• Your VAR Model is Secretly an Efficient and Explainable Generative Classifier
• Compact Autoregressive Network
• Designing ReLU Generative Networks to Enumerate Trees with a Given Tree Edit Distance
• A Review of Learning with Deep Generative Models from Perspective of Graphical Modeling
• Towards Better & Faster Autoregressive Image Generation: From the Perspective of Entropy
• ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis
• Towards Real-World Deepfake Detection: A Diverse In-the-wild Dataset of Forgery Faces
• \bfD³QE: Learning Discrete Distribution Discrepancy-aware Quantization Error for Autoregressive-Generated Image Detection
• REAR: Rethinking Visual Autoregressive Models via Generator-Tokenizer Consistency Regularization
• GenAR: Next-Scale Autoregressive Generation for Spatial Gene Expression Prediction
• Avoiding Latent Variable Collapse With Generative Skip Models
• Deep Learning for Image Super-Resolution: A Survey
• SoftCFG: Uncertainty-guided Stable Guidance for Visual Autoregressive Model
• Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAE
• Can multi-label classification networks know what they don't know?
• EchoGen: Generating Visual Echoes in Any Scene via Feed-Forward Subject-Driven Auto-Regressive Model
• MARS: Sound Generation via Multi-Channel Autoregression on Spectrograms
• Neural Density Estimation and Likelihood-free Inference
• Scale-Wise VAR is Secretly Discrete Diffusion
• FastFlow: Unsupervised Anomaly Detection and Localization via 2D Normalizing Flows
• Learning Likelihoods with Conditional Normalizing Flows
• Challenges and Solutions in Selecting Optimal Lossless Data Compression Algorithms
• High Fidelity Face Manipulation with Extreme Poses and Expressions
• Harnessing Synthetic Data from Generative AI for Statistical Inference
• Generating and designing DNA with deep generative models
• Focal Frequency Loss for Image Reconstruction and Synthesis
• BatchTNMC: Efficient sampling of two-dimensional spin glasses using tensor network Monte Carlo
• Is It Certainly a Deepfake? Reliability Analysis in Detection & Generation Ecosystem
• From Restoration to Reconstruction: Rethinking 3D Gaussian Splatting for Underwater Scenes
• Structure-Fair Quantum Circuit Complexity: An Auditable Information-Theoretic Lower Bound
• Solar Forecasting with Causality: A Graph-Transformer Approach to Spatiotemporal Dependencies
• Computer-Aided Design as Language
• EatGAN: An Edge-Attention Guided Generative Adversarial Network for Single Image Super-Resolution
• Color Visual Illusions: A Statistics-based Computational Model
• DOI: Divergence-based Out-of-Distribution Indicators via Deep Generative Models
• A Survey of Deep Learning for Scientific Discovery
• TSIT: A Simple and Versatile Framework for Image-to-Image Translation
• Transformer ASR with Contextual Block Processing
• MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space
• Diagnosing and Enhancing VAE Models
• Label-Noise Robust Generative Adversarial Networks
• On Self Modulation for Generative Adversarial Networks
• Stochastic Neural Network with Kronecker Flow
• Scalable Training for Vector-Quantized Networks with 100% Codebook Utilization
• SEEC: Segmentation-Assisted Multi-Entropy Models for Learned Lossless Image Compression
• UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation
• On the Evaluation of Conditional GANs
• MelNet: A Generative Model for Audio in the Frequency Domain
• Image-to-Image Translation: Methods and Applications
• Learning Scalable ℓ_∞-constrained Near-lossless Image Compression via Joint Lossy Image and Residual Compression
• A review of radar-based nowcasting of precipitation and applicable machine learning techniques
• Bit-Swap: Recursive Bits-Back Coding for Lossless Compression with Hierarchical Latent Variables
• Learning to Manipulate Deformable Objects without Demonstrations
• Jukebox: A Generative Model for Music
• Distillation of a tractable model from the VQ-VAE
• Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules
• Bigeminal Priors Variational auto-encoder
• Learning compact generalizable neural representations supporting perceptual grouping
• Deep Verifier Networks: Verification of Deep Discriminative Models with Deep Generative Models
• Leveraging Discriminative Latent Representations for Conditioning GAN-Based Speech Enhancement
• Likelihood Ratios for Out-of-Distribution Detection
• Fractal Flow: Hierarchical and Interpretable Normalizing Flow via Topic Modeling and Recursive Strategy
• BachGAN: High-Resolution Image Synthesis from Salient Object Layout
• Unsupervised Learning Layers for Video Analysis
• Deep Latent-Variable Kernel Learning
• SATURN: Autoregressive Image Generation Guided by Scene Graphs
• Symmetry-adapted generation of 3d point sets for the targeted discovery of molecules
• Learned Lossless Image Compression with a HyperPrior and Discretized Gaussian Mixture Likelihoods
• Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes
• From Classical Probabilistic Latent Variable Models to Modern Generative AI: A Unified Perspective
• Conditional Sound Generation Using Neural Discrete Time-Frequency Representation Learning
• A Survey of Black-Box Adversarial Attacks on Computer Vision Models
• Conditional Sum-Product Networks: Imposing Structure on Deep\n Probabilistic Architectures
• Generative Counterfactuals for Neural Networks via Attribute-Informed Perturbation
• Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis
• Neural quantum states for emitter dynamics in waveguide QED
• CDVAE: Co-embedding Deep Variational Auto Encoder for Conditional Variational Generation
• DCoAR: Deep Concept Injection into Unified Autoregressive Models for Personalized Text-to-Image Generation
• Learning to Generate 3D Shapes with Generative Cellular Automata
• Physical Layer Authentication Based on Hierarchical Variational Auto-Encoder for Industrial Internet of Things
• Data Priming Network for Automatic Check-Out
• Transflow Learning: Repurposing Flow Models Without Retraining
• Dueling Decoders: Regularizing Variational Autoencoder Latent Spaces
• Learning Content-Weighted Deep Image Compression
• Learning and Evaluating Representations for Deep One-class Classification
• Probabilistic Semantic Inpainting with Pixel Constrained CNNs
• The Six Fronts of the Generative Adversarial Networks
• Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?
• Densely connected normalizing flows
• Flow Stochastic Segmentation Networks
• A Comprehensive Review of Diffusion Models in Smart Agriculture: Progress, Applications, and Challenges
• Enhancing Scene Transition Awareness in Video Generation via Post-Training
• Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trends
• Variational Monocular Depth Estimation for Reliability Prediction
• EdiBERT, a generative model for image editing
• Orthogonal Convolutional Neural Networks
• Generative Models for Security: Attacks, Defenses, and Opportunities
• Hierarchical Adversarially Learned Inference
• Towards Robust Neural Vocoding for Speech Generation: A Survey
• A Forest from the Trees: Generation through Neighborhoods
• Sinusoidal Flow: A Fast Invertible Autoregressive Flow
• Triple Generative Adversarial Networks
• Controllable Video Generation: A Survey
• Locally Masked Convolution for Autoregressive Models
• Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction
• Speaker-adaptive neural vocoders for parametric speech synthesis systems
• Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Counts
• Self-Reflective Variational Autoencoder
• Can We Trust Deep Speech Prior?
• Semantic Image Manipulation Using Scene Graphs
• Context-aware Padding for Semantic Segmentation
• Implicit Autoencoders
• Variational Conditional GAN for Fine-grained Controllable Image Generation
• Taming generative video models for zero-shot optical flow extraction
• Data Depth as a Risk
• Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Image Generation
• Computationally Efficient Information-Driven Optical Design with Interchanging Optimization
• Image Generation from Layout
• Taming Data Challenges in ML-based Security Tasks Using Generative AI
• Future Data Helps Training: Modeling Future Contexts for Session-based Recommendation
• Lossless Image Compression Using a Multi-Scale Progressive Statistical Model
• Discriminator Contrastive Divergence: Semi-Amortized Generative Modeling by Exploring Energy of the Discriminator
• SemanticAdv: Generating Adversarial Examples via Attribute-conditional Image Editing
• Classification Accuracy Score for Conditional Generative Models
• Hita: Holistic Tokenizer for Autoregressive Image Generation
• LATTE: Latent Trajectory Embedding for Diffusion-Generated Image Detection

Related

• cFineGAN: Unsupervised multi-conditional fine-grained image generation
• PixelNN: Example-based Image Synthesis
• PixelCNN Models with Auxiliary Variables for Natural Image Modeling
• cGANs with Conditional Convolution Layer
• Conditional Invertible Neural Networks for Diverse Image-to-Image Translation