FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models

2018/10/02 by Will Grathwohl, Grathwohl, Will, Ricky T. Q. Chen +7 · 161 citations
Computer Science · Mathematics · Physics and Astronomy · #Computational Physics and Python Applications #Generative Adversarial Networks and Image Synthesis #Model Reduction and Neural Networks #cs.CV #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1810.01367

8 Pages, 6 figures

arxiv created 2018/10/22 · arxiv updated 2018/10/23

Abstract

A promising class of generative models maps points from a simple distribution to a complex distribution through an invertible neural network. Likelihood-based training of these models requires restricting their architectures to allow cheap computation of Jacobian determinants. Alternatively, the Jacobian trace can be used if the transformation is specified by an ordinary differential equation. In this paper, we use Hutchinson's trace estimator to give a scalable unbiased estimate of the log-density. The result is a continuous-time invertible generative model with unbiased density estimation and one-pass sampling, while allowing unrestricted neural network architectures. We demonstrate our approach on high-dimensional density estimation, image generation, and variational inference, achieving the state-of-the-art among exact likelihood methods with efficient sampling.

Cited by

• Joint Flow Matching for Generator-Consistent Classification
• All in One: Generative Modeling as Mean-Field Game Design
• Adaptive Probability Flow Residual Minimization for High-Dimensional Fokker-Planck Equations
• Robust and scalable simulation-based inference for gravitational wave signals with gaps
• Is There a Better Source Distribution than Gaussian? Exploring Source Distributions for Image Flow Matching
• Bidirectional Normalizing Flow: From Data to Noise and Back
• FALCON: Few-step Accurate Likelihoods for Continuous Flows
• One-Step Diffusion Samplers via Self-Distillation and Deterministic Flow
• Virtually Unrolling the Herculaneum Papyri by Diffeomorphic Spiral Fitting
• Exact and Parametric Dynamical System Representation of Nonlinear Functions
• Joint Distillation for Fast Likelihood Evaluation and Sampling in Flow-based Models
• DiG-Flow: Discrepancy-Guided Flow Matching for Robust VLA Models
• Generative models for crystalline materials
• Dynamical Implicit Neural Representations
• Latent Diffusion Inversion Requires Understanding the Latent Space
• VeCoR -- Velocity Contrastive Regularization for Flow Matching
• Breaking the Likelihood-Quality Trade-off in Diffusion Models by Merging Pretrained Experts
• Saving Foundation Flow-Matching Priors for Inverse Problems
• FlowPath: Learning Data-Driven Manifolds with Invertible Flows for Robust Irregularly-sampled Time Series Classification
• Synthesizing speech with selected perceptual voice qualities - A case study with creaky voice
• Learning Paths for Dynamic Measure Transport: A Control Perspective
• A Non-Adversarial Approach to Idempotent Generative Modelling
• Curly Flow Matching for Learning Non-gradient Field Dynamics
• Physics-informed continuous normalizing flows to learn the electric field within a time-projection chamber
• A global convergence theory for deep ReLU implicit networks via over-parameterization
• Normalizing flows for novelty detection in industrial time series data
• Discretize-Optimize vs. Optimize-Discretize for Time-Series Regression and Continuous Normalizing Flows
• Denoising Diffusion Probabilistic Models
• Neural Spline Flows
• Invertible Generative Modeling using Linear Rational Splines
• Learning Gradient Fields for Shape Generation
• Block Neural Autoregressive Flow
• Mixed Precision Training of Neural ODEs
• Variational Polya Tree
• On the flow matching interpretability
• Learning Discrete Distributions by Dequantization
• Video Consistency Distance: Enhancing Temporal Consistency for Image-to-Video Generation via Reward-Based Fine-Tuning
• Rethinking ResNets: Improved Stacking Strategies With High Order Schemes
• PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows
• LeapFactual: Reliable Visual Counterfactual Explanation Using Conditional Flow Matching
• Y-Shaped Generative Flows
• Moser Flow: Divergence-based Generative Modeling on Manifolds
• Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Models
• Score-Based Density Estimation from Pairwise Comparisons
• Integer Discrete Flows and Lossless Compression
• Relaxing Bijectivity Constraints with Continuously Indexed Normalising Flows
• Rectified-CFG++ for Flow Based Models
• Application of deep neural networks for computing the renormalization group flow of the two-dimensional phi⁴ field theory
• PointNSP: Autoregressive 3D Point Cloud Generation with Next-Scale Level-of-Detail Prediction
• Riemannian Consistency Model
• Invertible DenseNets with Concatenated LipSwish
• Flow Matching with Semidiscrete Couplings
• Let Physics Guide Your Protein Flows: Topology-aware Unfolding and Generation
• Hamiltonian Generative Networks
• Neural Density Estimation and Likelihood-free Inference
• Neural SDEs as Infinite-Dimensional GANs
• Random Projection Flows for Efficient Manifold Density Estimation
• Learning Likelihoods with Conditional Normalizing Flows
• Convex Potential Flows: Universal Probability Distributions with Optimal Transport and Convex Optimization
• Training Invertible Linear Layers through Rank-One Perturbations
• Flow-Induced Diagonal Gaussian Processes
• Gradient Boosted Normalizing Flows
• Task-agnostic Continual Learning with Hybrid Probabilistic Models
• CDFlow: Generative Gradient Flows for Configuration Space Distance Fields via Neural ODEs
• A Survey of Deep Learning-based Point Cloud Denoising
• Learning Neural Generative Dynamics for Molecular Conformation Generation
• Diffusion Secant Alignment for Score-Based Density Ratio Estimation
• Causal Navigation by Continuous-time Neural Networks
• Understanding and Mitigating Exploding Inverses in Invertible Neural Networks
• Diffusion Normalizing Flow
• Multiscale Deep Equilibrium Models
• Amortized Sampling with Transferable Normalizing Flows
• Mining gold from implicit models to improve likelihood-free inference
• Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes
• Observable Optimization for Precision Theory: Machine Learning Energy Correlators
• Reverse Convolution and Its Applications to Image Restoration
• Sliced Iterative Normalizing Flows
• Estimating Musical Surprisal from Audio in Autoregressive Diffusion Model Noise Spaces
• Flow Matching for Probabilistic Learning of Dynamical Systems from Missing or Noisy Data
• Flow Contrastive Estimation of Energy-Based Models
• Weighted Conditional Flow Matching
• MedSymmFlow: Bridging Generative Modeling and Classification in Medical Imaging through Symmetrical Flow Matching
• Large-time asymptotics in deep learning
• Optimal Unsupervised Domain Translation
• Learning Equivariant Energy Based Models with Equivariant Stein Variational Gradient Descent
• Learning Implicit Generative Models with Theoretical Guarantees
• Copula-Based Normalizing Flows
• Score-Based Generative Modeling through Stochastic Differential Equations
• Graphical Normalizing Flows
• InfoCNF: An Efficient Conditional Continuous Normalizing Flow with Adaptive Solvers
• NanoFlow: Scalable Normalizing Flows with Sublinear Parameter Complexity
• Learning more expressive joint distributions in multimodal variational methods
• Residual Flows for Invertible Generative Modeling
• Testing for Typicality with Respect to an Ensemble of Learned Distributions
• Implicit Machine Learning Force Fields Accelerate Molecular Dynamics Simulations
• Normalizing Flow to Augmented Posterior: Conditional Density Estimation with Interpretable Dimension Reduction for High Dimensional Data
• DexVLG: Dexterous Vision-Language-Grasp Model at Scale
• BoltzNCE: Learning Likelihoods for Boltzmann Generation with Stochastic Interpolants and Noise Contrastive Estimation
• Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows
• "Hey, that's not an ODE": Faster ODE Adjoints via Seminorms
• Towards foundational LiDAR world models with efficient latent flow matching
• 3D Shape Generation: A Survey
• Review: Ordinary Differential Equations For Deep Learning
• Distilling Normalizing Flows
• Control and optimization for Neural Partial Differential Equations in Supervised Learning
• Simulation-Free Differential Dynamics through Neural Conservation Laws
• Time-dependent density estimation using binary classifiers
• NeuralPDR: Neural Differential Equations as surrogate models for Photodissociation Regions
• Latent Neural Differential Equations for Video Generation
• Exponential Family Estimation via Adversarial Dynamics Embedding
• Audio synthesizer inversion in symmetric parameter spaces with approximately equivariant flow matching
• Neural Manifold Ordinary Differential Equations
• The Lipschitz Constant of Self-Attention
• Symmetrical Flow Matching: Unified Image Generation, Segmentation, and Classification with Score-Based Generative Models
• STARFlow: Scaling Latent Normalizing Flows for High-resolution Image Synthesis
• Simulating Fokker-Planck equations via mean field control of score-based normalizing flows
• Aligning Latent Spaces with Flow Priors
• Progressive Tempering Sampler with Diffusion
• On Fitting Flow Models with Large Sinkhorn Couplings
• Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models
• Mitigating mode collapse in normalizing flows by annealing with an adaptive schedule: Application to parameter estimation
• Large deviations for scaled families of Schrödinger bridges with reflection
• A Triangular Network For Density Estimation
• Neural representation and generation for RNA secondary structures
• Learning High Dimensional Wasserstein Geodesics
• Ergodic Generative Flows
• FOLIAGE: Towards Physical Intelligence World Models Via Unbounded Surface Evolution
• JAPAN: Joint Adaptive Prediction Areas with Normalising-Flows
• CaSPR: Learning Canonical Spatiotemporal Point Cloud Representations
• Streaming Flow Policy: Simplifying diffusion/flow-matching policies by treating action trajectories as flow trajectories
• Symplectic Generative Networks (SGNs): A Hamiltonian Framework for Invertible Deep Generative Modeling
• Continuous Graph Flow
• Learning ODEs via Diffeomorphisms for Fast and Robust Integration
• Efficient and Unbiased Sampling from Boltzmann Distributions via Variance-Tuned Diffusion Models
• Bilateral Denoising Diffusion Models
• On the Relation between Rectified Flows and Optimal Transport
• Self Normalizing Flows
• Continuous Normalizing Flows for Uncertainty-Aware Human Pose Estimation
• Gradients as Features for Deep Representation Learning
• Beyond Equilibrium: Non-Equilibrium Foundations Should Underpin Generative Processes in Complex Dynamical Systems
• Generative Learning With Euler Particle Transport
• Liouville PDE-based sliced-Wasserstein flow for fair regression
• Accelerating Continuous Normalizing Flow with Trajectory Polynomial Regularization
• Oh SnapMMD! Forecasting Stochastic Dynamics Beyond the Schrödinger Bridge's End
• Closing the Dequantization Gap: PixelCNN as a Single-Layer Flow
• Inferring stochastic dynamics with growth from cross-sectional data
• CacheFlow: Fast Human Motion Prediction by Cached Normalizing Flow
• Multi-View Wireless Sensing via Conditional Generative Learning: Framework and Model Design
• CN101 - A Digital Thermodynamic Computer for Generative AI
• Path Gradients after Flow Matching
• Neural Variational Gradient Descent
• Semantic and Geometric Unfolding of StyleGAN Latent Space
• Good Things Come in Pairs: Paired Autoencoders for Inverse Problems
• Learning the Simplest Neural ODE
• PackFlow: Generative Molecular Crystal Structure Prediction via Reinforcement Learning Alignment
• Generative Replica-Exchange: A Flow-based Framework for Accelerating Replica Exchange Simulations
• JFlow: Model-Independent Spherical Jeans Analysis using Equivariant Continuous Normalizing Flows
• ADiff4TPP: Asynchronous Diffusion Models for Temporal Point Processes
• Wavelet Flow: Fast Training of High Resolution Normalizing Flows
• Robust Learning with Implicit Residual Networks
• Dysarthria Normalization via Local Lie Group Transformations for Robust ASR
• Potential Matching Optimal Transport: Continuous Normalizing Flows for Exact p-Wasserstein Dynamics
• Hierarchical Flow Matching for 3D Point Cloud Generation

Related

• Residual Flows for Invertible Generative Modeling
• Learning Generative Models using Denoising Density Estimators
• Diffusion-Model-Assisted Supervised Learning of Generative Models for Density Estimation
• Adaptive Density Estimation for Generative Models
• Can Push-forward Generative Models Fit Multimodal Distributions?