Flow Matching Guide and Code

2024/12/09 by Yaron Lipman, Marton Havasi, Lipman, Yaron +18 · 10 voices · 80 citations
Decision Sciences · Computer Science · #Simulation Techniques and Applications #Software System Performance and Reliability

paper · pdf · doi:10.48550/arxiv.2412.06264

Abstract

Flow Matching (FM) is a recent framework for generative modeling that has achieved state-of-the-art performance across various domains, including image, video, audio, speech, and biological structures. This guide offers a comprehensive and self-contained review of FM, covering its mathematical foundations, design choices, and extensions. By also providing a PyTorch package featuring relevant examples (e.g., image and text generation), this work aims to serve as a resource for both novice and experienced researchers interested in understanding, applying and further developing FM.

Cited by

• ConFlow: Constraints-Guided Learning with Flow Matching for Motion Generation
• Native Extrapolation Awareness in Flow-Based Conditional Generation
• TVGL-CFM:Generating and Forecasting Time-Varying Trajectories of Dynamic Networks with Conditional Flow Matching
• Trajectory-Aware Flow Matching for Topology Optimisation
• Consistent Sampling and Simulation: Molecular Dynamics with Energy-Based Diffusion Models
• On the Closed-Form of Flow Matching: Generalization Does Not Arise from Target Stochasticity
• High-Performance Self-Supervised Learning by Joint Training of Flow Matching
• Diffusion Knows Transparency: Repurposing Video Diffusion for Transparent Object Depth and Normal Estimation
• Energy-Guided Flow Matching Enables Few-Step Conformer Generation and Ground-State Identification
• Generative Modeling by Value-Driven Transport
• Robust and scalable simulation-based inference for gravitational wave signals with gaps
• BézierFlow: Learning Bézier Stochastic Interpolant Schedulers for Few-Step Generation
• Exploring the Design Space of Transition Matching
• REST: Diffusion-based Real-time End-to-end Streaming Talking Head Generation via ID-Context Caching and Asynchronous Streaming Distillation
• Investigating training objective for flow matching-based speech enhancement
• OCCDiff: Occupancy Diffusion Model for High-Fidelity 3D Building Reconstruction from Noisy Point Clouds
• Scalable Offline Model-Based RL with Action Chunks
• Sample from What You See: Visuomotor Policy Learning via Diffusion Bridge with Observation-Embedded Stochastic Differential Equation
• WAM-Flow: Parallel Coarse-to-Fine Motion Planning via Discrete Flow Matching for Autonomous Driving
• LatentFM: A Latent Flow Matching Approach for Generative Medical Image Segmentation
• Deep Unsupervised Anomaly Detection in Brain Imaging: Large-Scale Benchmarking and Bias Analysis
• Generative Anchored Fields: Controlled Data Generation via Emergent Velocity Fields and Transport Algebra
• Flow Density Control: Generative Optimization Beyond Entropy-Regularized Fine-Tuning
• FlexiFlow: decomposable flow matching for generation of flexible molecular ensemble
• SPEAR-1: Scaling Beyond Robot Demonstrations via 3D Understanding
• Time dependent loss reweighting for flow matching and diffusion models is theoretically justified
• Saving Foundation Flow-Matching Priors for Inverse Problems
• FUSE: A Flow-based Mapping Between Shapes
• Tensor Gauge Flow Models
• Learning Straight Flows: Variational Flow Matching for Efficient Generation
• Arcee: Differentiable Recurrent State Chain for Generative Vision Modeling with Mamba SSMs
• Branching Flows: Discrete, Continuous, and Manifold Flow Matching with Splits and Deletions
• Learning by Neighbor-Aware Semantics, Deciding by Open-form Flows: Towards Robust Zero-Shot Skeleton Action Recognition
• On Flow Matching KL Divergence
• Multi-agent Coordination via Flow Matching
• SyMuPe: Affective and Controllable Symbolic Music Performance
• Simplex-to-Euclidean Bijections for Categorical Flow Matching
• Distance Marching for Generative Modeling
• FragmentFlow: Scalable Transition State Generation for Large Molecules
• Generative Bayesian Optimization: Generative Models as Acquisition Functions
• EnzyControl: Adding Functional and Substrate-Specific Control for Enzyme Backbone Generation
• Sequence Modeling with Spectral Mean Flows
• Transitive RL: Value Learning via Divide and Conquer
• Blockwise Flow Matching: Improving Flow Matching Models For Efficient High-Quality Generation
• Coupled Diffusion Sampling for Training-Free Multi-View Image Editing
• Adapting Noise to Data: Generative Flows from 1D Processes
• Equilibrium Matching: Generative Modeling with Implicit Energy-Based Models
• Flow Matching-Based Autonomous Driving Planning with Advanced Interactive Behavior Modeling
• X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Model
• Failure Prediction at Runtime for Generative Robot Policies
• Expressive Value Learning for Scalable Offline Reinforcement Learning
• Value Flows
• Carré du champ flow matching: better quality-generalisation tradeoff in generative models
• OneFlow: Concurrent Mixed-Modal and Interleaved Generation with Edit Flows
• MoReFlow: Motion Retargeting Learning through Unsupervised Flow Matching
• Flow Matching with Semidiscrete Couplings
• Let Physics Guide Your Protein Flows: Topology-aware Unfolding and Generation
• Chance-constrained Flow Matching for High-Fidelity Constraint-aware Generation
• MSG: Multi-Stream Generative Policies for Sample-Efficient Robotic Manipulation
• GLASS Flows: Transition Sampling for Alignment of Flow and Diffusion Models
• Diffusion Bridge or Flow Matching? A Unifying Framework and Comparative Analysis
• Stochastic Interpolants via Conditional Dependent Coupling
• Error Analysis of Discrete Flow with Generator Matching
• FS-DFM: Fast and Accurate Long Text Generation with Few-Step Diffusion Language Models
• DynaFlow: Dynamics-embedded Flow Matching for Physically Consistent Motion Generation from State-only Demonstrations
• FinFlowRL: An Imitation-Reinforcement Learning Framework for Adaptive Stochastic Control in Finance
• Dynamic Objects Relocalization in Changing Environments with Flow Matching
• Online Dose Verification in VHEE Radiotherapy Using Bremsstrahlung Radiation
• Energy-Weighted Flow Matching: Unlocking Continuous Normalizing Flows for Efficient and Scalable Boltzmann Sampling
• Align-Then-stEer: Adapting the Vision-Language Action Models through Unified Latent Guidance
• Multimodal Generative Flows for LHC Jets
• FinFlowRL: An Imitation-Reinforcement Learning Framework for Adaptive Stochastic Control in Finance
• MedShift: Implicit Conditional Transport for X-Ray Domain Adaptation
• Flowing Straighter with Conditional Flow Matching for Accurate Speech Enhancement
• Energy-Based Flow Matching for Generating 3D Molecular Structure
• Follow My Hold: Hand-Object Interaction Reconstruction through Geometric Guidance
• It's not a FAD: first results in using Flows for unsupervised Anomaly Detection at 40 MHz at the Large Hadron Collider
• OM2P: Offline Multi-Agent Mean-Flow Policy
• CloudBreaker: Breaking the Cloud Covers of Sentinel-2 Images using Multi-Stage Trained Conditional Flow Matching on Sentinel-1
• FMPlug: Plug-In Foundation Flow-Matching Priors for Inverse Problems

Discussions

• Inventors of flow matching have released a comprehensive guide going over the math & code of flow matching! Also covers variants like non-Euclidean & discrete flow matching. A PyTorch library is als [bsky, 109 points, 1 comments]
• Flow Matching Guide and Code arxiv.org/abs/2412.06264 [bsky, 24 points, 0 comments]
• What a week/month/year it has been for flow matching! The team from FLAIR released an extended version of some of their FM articles. As a bonus, they also released the associated code (for original, d [bsky, 20 points, 0 comments]
• Tutorial on flow matching today at @neuripsconf.bsky.social. I'll miss it 😿. The presenters released a very comprehensive document+code+library on flow matching arxiv.org/pdf/2412.06264 Need visual [bsky, 4 points, 0 comments]
• Flow Matching Guide and Code [hn, 3 points, 0 comments]
• All you need to know about flow matching Comprehensive review and pytorch code about flow matching and related techniques in both discrete and continuous spaces arxiv.org/abs/2412.06264 code on g [bsky, 3 points, 0 comments]
• Over the past few months, I have loved learning about Flow Matching (FM) models. With groups starting to apply FM/Diffusion models to neuroscience, this recent review on FM by Meta FAIR is a great res [bsky, 2 points, 1 comments]
• An 80 page description of flow matching, with pytorch examples. From Meta's FAIR lab. #machinelearning https://arxiv.org/abs/2412.06264 [bsky, 1 points, 0 comments]
• Flow Matching got an impressive guide & tutorial today (by FAIR @ Meta). Credits to Ricky Chen et al. 📒Flow Matching basics & advanced arxiv.org/abs/2412.06264 👨‍💻 Code facebookresearch.github.io [bsky, 1 points, 0 comments]
• Another long read, but an essential one for those needing to know more about flow matching arxiv.org/pdf/2412.06264 [bsky, 0 points, 1 comments]

Related

• Flow Generator Matching
• Explicit Flow Matching: On The Theory of Flow Matching Algorithms with Applications
• Blockwise Flow Matching: Improving Flow Matching Models For Efficient High-Quality Generation
• Flow Matching Meets Biology and Life Science: A Survey
• INRFlow: Flow Matching for INRs in Ambient Space