Latent Normalizing Flows for Discrete Sequences

2019/01/29 by Zachary M. Ziegler, Alexander M. Rush, Ziegler, Zachary M. +1 · 17 citations
Computer Science · Mathematics · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Music and Audio Processing #Natural Language Processing Techniques #Topic Modeling #cs.CL #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1901.10548

openalex publication_date 2019/01/29 · arxiv created 2019/06/04 · arxiv updated 2019/06/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Normalizing flows are a powerful class of generative models for continuous random variables, showing both strong model flexibility and the potential for non-autoregressive generation. These benefits are also desired when modeling discrete random variables such as text, but directly applying normalizing flows to discrete sequences poses significant additional challenges. We propose a VAE-based generative model which jointly learns a normalizing flow-based distribution in the latent space and a stochastic mapping to an observed discrete space. In this setting, we find that it is crucial for the flow-based distribution to be highly multimodal. To capture this property, we propose several normalizing flow architectures to maximize model flexibility. Experiments consider common discrete sequence tasks of character-level language modeling and polyphonic music generation. Our results indicate that an autoregressive flow-based model can match the performance of a comparable autoregressive baseline, and a non-autoregressive flow-based model can improve generation speed with a penalty to performance.

Cited by

• Mapping of Lesion Images to Somatic Mutations
• Copula Flows for Synthetic Data Generation
• BERT as a Teacher: Contextual Embeddings for Sequence-Level Reward
• Discrete Flows: Invertible Generative Models of Discrete Data
• Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows
• Variational Dynamic Mixtures
• Intensity-Free Learning of Temporal Point Processes
• Conditional Flow Variational Autoencoders for Structured Sequence Prediction
• Causal Mediation Analysis with Multiple Mediators: A Simulation Approach
• Scaling Probabilistic Circuits via Monarch Matrices
• Discrete Markov Bridge
• APo-VAE: Text Generation in Hyperbolic Space
• Learning Flexible Forward Trajectories for Masked Molecular Diffusion
• CtrlDiff: Boosting Large Diffusion Language Models with Dynamic Block Prediction and Controllable Generation
• Latent Reasoning with Normalizing Flows
• Generalized Discrete Diffusion from Snapshots
• Latent Bayesian Optimization via Autoregressive Normalizing Flows

Related

• Discrete Flows: Invertible Generative Models of Discrete Data
• Generative Latent Flow
• Discrete Flow Maps
• Normalizing Flows are Capable Generative Models
• Normalizing Flow with Variational Latent Representation