Densely connected normalizing flows

2021/06/08 by Matej Grcić, Grcić, Matej, Ivan Grubišić +3 · 3 citations
Computer Science · Medicine · #Advanced Neuroimaging Techniques and Applications #Artificial Intelligence (cs.AI) #Computer Graphics and Visualization Techniques #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #cs.AI #cs.CV #cs.LG

paper · pdf · doi:10.48550/arxiv.2106.04627

Accepted at NeurIPS2021

openalex publication_date 2021/06/08 · arxiv created 2021/11/02 · arxiv updated 2021/11/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Normalizing flows are bijective mappings between inputs and latent representations with a fully factorized distribution. They are very attractive due to exact likelihood valuation and efficient sampling. However, their effective capacity is often insufficient since the bijectivity constraint limits the model width. We address this issue by incrementally padding intermediate representations with noise. We precondition the noise in accordance with previous invertible units, which we describe as cross-unit coupling. Our invertible glow-like modules increase the model expressivity by fusing a densely connected block with Nystrom self-attention. We refer to our architecture as DenseFlow since both cross-unit and intra-module couplings rely on dense connectivity. Experiments show significant improvements due to the proposed contributions and reveal state-of-the-art density estimation under moderate computing budgets.

Citations

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Auto-Encoding Variational Bayes
• Training Products of Experts by Minimizing Contrastive Divergence
• ImageNet Large Scale Visual Recognition Challenge
• Pixel Recurrent Neural Networks
• Improving Variational Inference with Inverse Autoregressive Flow
• Improved Training of Wasserstein GANs
• Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design
• Conditional Image Generation with PixelCNN Decoders
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications
• Quantifying the Carbon Emissions of Machine Learning
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Generative adversarial networks
• Learning Multiple Layers of Features from Tiny Images
• Invertible DenseNets
• Invertible DenseNets with Concatenated LipSwish
• Variational Diffusion Models

Cited by

• Breaking the Likelihood-Quality Trade-off in Diffusion Models by Merging Pretrained Experts
• A Data-Driven Novelty Score for Diverse In-Vehicle Data Recording
• Accelerate TarFlow Sampling with GS-Jacobi Iteration

Related

• Variations and Relaxations of Normalizing Flows
• NanoFlow: Scalable Normalizing Flows with Sublinear Parameter Complexity
• Attentive Contractive Flow with Lipschitz-constrained Self-Attention
• Embedded-model flows: Combining the inductive biases of model-free deep learning and explicit probabilistic modeling
• You say Normalizing Flows I see Bayesian Networks