Simple and Effective Masked Diffusion Language Models

2024/06/11 by Subham Sekhar Sahoo, Marianne Arriola, Sahoo, Subham Sekhar +13 · 277 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2406.07524

openalex publication_date 2024/06/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

While diffusion models excel at generating high-quality images, prior work reports a significant performance gap between diffusion and autoregressive (AR) methods in language modeling. In this work, we show that simple masked discrete diffusion is more performant than previously thought. We apply an effective training recipe that improves the performance of masked diffusion models and derive a simplified, Rao-Blackwellized objective that results in additional improvements. Our objective has a simple form -- it is a mixture of classical masked language modeling losses -- and can be used to train encoder-only language models that admit efficient samplers, including ones that can generate arbitrary lengths of text semi-autoregressively like a traditional language model. On language modeling benchmarks, a range of masked diffusion models trained with modern engineering practices achieves a new state-of-the-art among diffusion models, and approaches AR perplexity. We provide the code, along with a blog post and video tutorial on the project page: https://s-sahoo.com/mdlm

Cited by

• UNIFUSION: Adapting Autoregressive Language Models into Discrete Diffusion under a Unified Reverse-Rate Objective
• dUltra: Ultra-Fast Diffusion Language Models via Reinforcement Learning
• On the Role of Discreteness in Diffusion LLMs
• Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion Language Model Backbone
• Where Quality Breaks in Compressed Short-Text Generation: Staged Bottleneck Localization
• PreDiff-LM: Pretrained Discrete Masked Diffusion Language Modeling with Hybrid Attention
• TRE: Training-Free Hallucination Detection for Diffusion Language Models
• FRIGID: Scaling Diffusion-Based Molecular Generation from Mass Spectra at Training and Inference Time
• Optimizing Decoding Paths in Masked Diffusion Models by Quantifying Uncertainty
• PairFlow: Closed-Form Source-Target Coupling for Few-Step Generation in Discrete Flow Models
• Corrective Diffusion Language Models
• Dual-objective Language Models: Training Efficiency Without Overfitting
• ViewMask-1-to-3: Multi-View Consistent Image Generation via Multimodal Discrete Diffusion Models
• SDAR-VL: Stable and Efficient Block-wise Diffusion for Vision-Language Understanding
• Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speed
• Scaling Behavior of Discrete Diffusion Language Models
• Guided Transfer Learning for Discrete Diffusion Models
• Learning Unmasking Policies for Diffusion Language Models
• From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs
• Generative Recursive Reasoning
• Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspective
• dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoning
• MDiff4STR: Mask Diffusion Model for Scene Text Recognition
• MM-ACT: Learn from Multimodal Parallel Generation to Act
• Masked Diffusion for Generative Recommendation
• Masks Can Be Distracting: On Context Comprehension in Diffusion Language Models
• Beyond Confidence: Adaptive and Coherent Decoding for Diffusion Language Models
• Which Layer Causes Distribution Deviation? Entropy-Guided Adaptive Pruning for Diffusion and Flow Models
• From Bits to Rounds: Parallel Decoding with Exploration for Diffusion Language Models
• Orchestrating Dual-Boundaries: An Arithmetic Intensity Inspired Acceleration Framework for Diffusion Language Models
• Masked Diffusion Models are Secretly Learned-Order Autoregressive Models
• Demystifying Diffusion Objectives: Reweighted Losses are Better Variational Bounds
• Bringing Stability to Diffusion: Decomposing and Reducing Variance of Training Masked Diffusion Models
• WavefrontDiffusion: Dynamic Decoding Schedule for Improved Reasoning
• MDG: Masked Denoising Generation for Multi-Agent Behavior Modeling in Traffic Environments
• Branching Flows: Discrete, Continuous, and Manifold Flow Matching with Splits and Deletions
• KLASS: KL-Guided Fast Inference in Masked Diffusion Models
• Optimal Inference Schedules for Masked Diffusion Models
• Diffusion Language Models are Super Data Learners
• Lookahead Unmasking Elicits Accurate Decoding in Diffusion Language Models
• Effective Test-Time Scaling of Discrete Diffusion through Iterative Refinement
• SpecDiff-2: Scaling Diffusion Drafter Alignment For Faster Speculative Decoding
• Simplex-to-Euclidean Bijections for Categorical Flow Matching
• Masked Diffusion Captioning for Visual Feature Learning
• Don't Let It Fade: Preserving Edits in Diffusion Language Models via Token Timestep Allocation
• Error Bounds and Optimal Schedules for Masked Diffusions with Factorized Approximations
• From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models
• How to make the most of your masked language model for protein engineering
• Information-Theoretic Discrete Diffusion
• Variational Masked Diffusion Models
• Sampling from Energy distributions with Target Concrete Score Identity
• Simple Denoising Diffusion Language Models
• Encoder-Decoder Diffusion Language Models for Efficient Training and Inference
• CANDI: Hybrid Discrete-Continuous Diffusion Models
• Aligning Diffusion Language Models via Unpaired Preference Optimization
• Parallel Sampling from Masked Diffusion Models via Conditional Independence Testing
• Efficient semantic uncertainty quantification in language models via diversity-steered sampling
• From Masks to Worlds: A Hitchhiker's Guide to World Models
• Loopholing Discrete Diffusion: Deterministic Bypass of the Sampling Wall
• From Denoising to Refining: A Corrective Framework for Vision-Language Diffusion Model
• DiffGRM: Diffusion-based Generative Recommendation Model
• Latent-Augmented Discrete Diffusion Models
• Planned Diffusion
• Soft-Masked Diffusion Language Models
• Planner and Executor: Collaboration between Discrete Diffusion And Autoregressive Models in Reasoning
• Attention Is All You Need for KV Cache in Diffusion LLMs
• Inference-Time Search Using Side Information for Diffusion-Based Image Reconstruction
• Test-Time Anchoring for Discrete Diffusion Posterior Sampling
• UltraLLaDA: Scaling the Context Length to 128K for Diffusion Large Language Models
• On the Quantization Robustness of Diffusion Language Models in Coding Benchmarks
• SPG: Sandwiched Policy Gradient for Masked Diffusion Language Models
• Diffusion-Inspired Masked Fine-Tuning for Knowledge Injection in Autoregressive LLMs
• Guided Star-Shaped Masked Diffusion
• Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimization
• Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimization
• Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistency
• Next Semantic Scale Prediction via Hierarchical Diffusion Language Models
• DiFFPO: Training Diffusion LLMs to Reason Fast and Furious via Reinforcement Learning
• Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understanding
• SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation
• Discrete Diffusion Models with MLLMs for Unified Medical Multimodal Generation
• Improving Discrete Diffusion Unmasking Policies Beyond Explicit Reference Policies
• Membership Inference Attacks on Tokenizers of Large Language Models
• Fine-Tuning Masked Diffusion for Provable Self-Correction
• Finish First, Perfect Later: Test-Time Token-Level Cross-Validation for Diffusion Large Language Models
• ParallelBench: Understanding the Trade-offs of Parallel Decoding in Diffusion LLMs
• LaDiR: Latent Diffusion Enhances LLMs for Text Reasoning
• Demystifying MaskGIT Sampler and Beyond: Adaptive Order Selection in Masked Diffusion
• Drax: Speech Recognition with Discrete Flow Matching
• Rainbow Padding: Mitigating Early Termination in Instruction-Tuned Diffusion LLMs
• Self-Speculative Masked Diffusions
• OneFlow: Concurrent Mixed-Modal and Interleaved Generation with Edit Flows
• DMark: Order-Agnostic Watermarking for Diffusion Large Language Models
• Consolidating Reinforcement Learning for Multimodal Discrete Diffusion Models
• Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasoner
• Toward Safer Diffusion Language Models: Discovery and Mitigation of Priming Vulnerability
• Diffusion Alignment as Variational Expectation-Maximization
• Continuously Augmented Discrete Diffusion model for Categorical Generative Modeling
• AReUReDi: Annealed Rectified Updates for Refining Discrete Flows with Multi-Objective Guidance
• Free Draft-and-Verification: Toward Lossless Parallel Decoding for Diffusion Large Language Models
• dParallel: Learnable Parallel Decoding for dLLMs
• Refine Drugs, Don't Complete Them: Uniform-Source Discrete Flows for Fragment-Based Drug Discovery
• dVLA: Diffusion Vision-Language-Action Model with Multimodal Chain-of-Thought
• TraceDet: Hallucination Detection from the Decoding Trace of Diffusion Large Language Models
• RFG: Test-Time Scaling for Diffusion Large Language Model Reasoning with Reward-Free Guidance
• Watermarking Diffusion Language Models
• TR2-D2: Tree Search Guided Trajectory-Aware Fine-Tuning for Discrete Diffusion
• Deep Thinking by Markov Chain of Continuous Thoughts
• Ultra-Fast Language Generation via Discrete Diffusion Divergence Instruct
• Learning to Parallel: Accelerating Diffusion Large Language Models via Learnable Parallel Decoding
• DiffuTester: Accelerating Unit Test Generation for Diffusion LLMs via Mining Structural Pattern
• Taming Masked Diffusion Language Models via Consistency Trajectory Reinforcement Learning with Fewer Decoding Step
• Training Optimal Large Diffusion Language Models
• Electric Currents for Discrete Data Generation
• RIV: Recursive Introspection Mask Diffusion Vision Language Model
• Planner Aware Path Learning in Diffusion Language Models Training
• Entering the Era of Discrete Diffusion Models: A Benchmark for Schrödinger Bridges and Entropic Optimal Transport
• Tree Reward-Aligned Search for TReASURe in Masked Diffusion Language Models
• d²Cache: Accelerating Diffusion-Based LLMs via Dual Adaptive Caching
• Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action Spaces
• Soft-Di[M]O: Improving One-Step Discrete Image Generation with Soft Embeddings
• Scale-Wise VAR is Secretly Discrete Diffusion
• Decoding quantum low density parity check codes with diffusion
• Discrete Guidance Matching: Exact Guidance for Discrete Flow Matching
• Error Analysis of Discrete Flow with Generator Matching
• Enabling Approximate Joint Sampling in Diffusion LMs
• Deterministic Discrete Denoising
• Understanding and Enhancing Mask-Based Pretraining towards Universal Representations
• d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimation
• FS-DFM: Fast and Accurate Long Text Generation with Few-Step Diffusion Language Models
• From Text to Talk: Audio-Language Model Needs Non-Autoregressive Joint Training
• Discrete Diffusion for Generative Modeling of Text-Aligned Speech Tokens
• Learnable Sampler Distillation for Discrete Diffusion Models
• Where and When to Commit: Candidate-Aware Decoding for Diffusion Language Models
• Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation
• ELF: Embedded Language Flows
• Dream-Cubed: Controllable Generative Modeling in Minecraft by Training on Billions of Cubes
• Where-to-Unmask: Ground-Truth-Guided Unmasking Order Learning for Masked Diffusion Language Models
• Audio-Conditioned Diffusion LLMs for ASR and Deliberation Processing
• Monte Carlo Tree Diffusion with Multiple Experts for Protein Design
• Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning
• Masked Diffusion Models as Energy Minimization
• DSFT: Inspiring Diffusion Large Language Models to Comprehend Mathematical and Logical Patterns
• ShortListing Model: A Streamlined SimplexDiffusion for Discrete Variable Generation
• Beyond Autoregression: An Empirical Study of Diffusion Large Language Models for Code Generation
• Inpainting-Guided Policy Optimization for Diffusion Large Language Models
• LLaDA-VLA: Vision Language Diffusion Action Models
• Masked Diffusion Language Models with Frequency-Informed Training
• Dream-Coder 7B: An Open Diffusion Language Model for Code
• Any-Order Flexible Length Masked Diffusion
• Diffusion Language Models Know the Answer Before Decoding
• Blockwise SFT for Diffusion Language Models: Reconciling Bidirectional Attention and Autoregressive Decoding
• Scaffold Diffusion: Sparse Multi-Category Voxel Structure Generation with Discrete Diffusion
• Dream 7B: Diffusion Large Language Models
• Gener anno : A Genomic Foundation Model for Metagenomic Annotation
• ToolACE-MT: Non-Autoregressive Generation for Agentic Multi-Turn Interaction
• MDPO: Overcoming the Training-Inference Divide of Masked Diffusion Language Models
• Where to Start Alignment? Diffusion Large Language Model May Demand a Distinct Position
• Improving Text Style Transfer using Masked Diffusion Language Models with Inference-time Scaling
• Thinking Inside the Mask: In-Place Prompting in Diffusion LLMs
• MDNS: Masked Diffusion Neural Sampler via Stochastic Optimal Control
• Time Is a Feature: Exploiting Temporal Dynamics in Diffusion Language Models
• The Cosine Schedule is Fisher-Rao-Optimal for Masked Discrete Diffusion Models
• One Model for All: Unified Try-On and Try-Off in Any Pose via LLM-Inspired Bidirectional Tweedie Diffusion
• Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference
• LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understanding
• TreeDiff: AST-Guided Code Generation with Diffusion LLMs
• ReDi: Rectified Discrete Flow
• Diffusion Beats Autoregressive in Data-Constrained Settings
• Wide-In, Narrow-Out: Revokable Decoding for Efficient and Effective DLLMs
• DIFFA: Large Language Diffusion Models Can Listen and Understand
• ARTreeFormer: A Faster Attention-based Autoregressive Model for Phylogenetic Inference
• Language Models for Controllable DNA Sequence Design
• The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models
• The Serial Scaling Hypothesis
• SynCoGen: Synthesizable 3D Molecule Generation via Joint Reaction and Coordinate Modeling
• Why Masking Diffusion Works: Condition on the Jump Schedule for Improved Discrete Diffusion
• Theory-Informed Improvements to Classifier-Free Guidance for Discrete Diffusion Models
• Inference-Time Scaling of Diffusion Language Models with Particle Gibbs Sampling
• Token-Based Audio Inpainting via Discrete Diffusion
• Predicting and generating antibiotics against future pathogens with ApexOracle
• A Survey on Latent Reasoning
• wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models
• Discrete Diffusion Trajectory Alignment via Stepwise Decomposition
• Representations from Pretrained Machine-Learning Interatomic Potentials as Coarse Coordinates for Material Generation and Evaluation
• Predict-then-Diffuse: Adaptive Response Length for Compute-Budgeted Inference in Diffusion LLMs
• Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMs
• Iterative Distillation for Reward-Guided Fine-Tuning of Diffusion Models in Biomolecular Design
• Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows
• Edit Flows: Flow Matching with Edit Operations
• Cosmos: Compressed and Smooth Latent Space for Text Diffusion Modeling
• Diffusion Tree Sampling: Scalable inference-time alignment of diffusion models
• DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation
• Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture
• Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spaces
• Plan for Speed: Dilated Scheduling for Masked Diffusion Language Models
• LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMs
• DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretation
• Flexible-length Text Infilling for Discrete Diffusion Models
• Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing
• Discrete Diffusion in Large Language and Multimodal Models: A Survey
• Generative or Discriminative? Revisiting Text Classification in the Era of Transformers
• What Exactly Does Guidance Do in Masked Discrete Diffusion Models
• The Diffusion Duality
• Multiverse: Your Language Models Secretly Decide How to Parallelize and Merge Generation
• Escaping Confidence Trap: Evolutionary Decoding for Mathematical Reasoning in Diffusion LLMs
• Synthesis of discrete-continuous quantum circuits with multimodal diffusion models
• Esoteric Language Models: A Family of Any-Order Diffusion LLMs
• Absorb and Converge: Provable Convergence Guarantee for Absorbing Discrete Diffusion Models
• Neuro-Symbolic Generative Diffusion Models for Physically Grounded, Robust, and Safe Generation
• Accelerating Diffusion LLMs via Adaptive Parallel Decoding
• RelDiff: Relational Data Generative Modeling with Graph-Based Diffusion Models
• DLM-One: Diffusion Language Models for One-Step Sequence Generation
• Learning Distributions over Permutations and Rankings with Factorized Representations
• Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmasking
• Inference-time Scaling of Diffusion Models through Classical Search
• Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Model
• Inference-Time Scaling of Discrete Diffusion Models via Importance Weighting and Optimal Proposal Design
• Efficient Controllable Diffusion via Optimal Classifier Guidance
• Breaking AR's Sampling Bottleneck: Provable Acceleration via Diffusion Language Models
• FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusion
• ReDDiT: Rehashing Noise for Discrete Visual Generation
• Adaptive Classifier-Free Guidance via Dynamic Low-Confidence Masking
• FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocities
• LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models
• A Framework for Stochastic Differentiable Programming
• Anchored Diffusion Language Model
• Partition Generative Modeling: Masked Modeling Without Masks
• Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Masking
• Discrete Neural Flow Samplers with Locally Equivariant Transformer
• Variational Autoencoding Discrete Diffusion with Enhanced Dimensional Correlations Modeling
• Alignment and Safety of Diffusion Models via Reinforcement Learning and Reward Modeling: A Survey
• Bigger Isn't Always Memorizing: Early Stopping Overparameterized Diffusion Models
• Learning Flexible Forward Trajectories for Masked Molecular Diffusion
• LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning
• From Prediction to Perfection: Introducing Refinement to Autoregressive Image Generation
• dKV-Cache: The Cache for Diffusion Language Models
• CtrlDiff: Boosting Large Diffusion Language Models with Dynamic Block Prediction and Controllable Generation
• dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Caching
• Exploring More to Solve More: Boosting Diversity in Text Diffusion Models via Entropy-Based Guidance
• Plug-and-Play Guidance for Discrete Diffusion Models via Gradient-Informed Logit Correction
• HERALD: High-Throughput Block Diffusion LLM Serving via CPU-GPU Cooperative KV Cache Retrieval
• Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metrics
• DLLM-TTS: Block Discrete Diffusion Language Model for Text-to-Speech Synthesis
• Multi-Objective-Guided Discrete Flow Matching for Controllable Biological Sequence Design
• Insertion Language Models: Sequence Generation with Arbitrary-Position Insertions
• Expert-Choice Routing Enables Adaptive Computation in Diffusion Language Models
• A Comparative analysis of Layer-wise Representational Capacity in AR and Diffusion LLMs
• Just on Time: Token-Level Early Stopping for Diffusion Language Models
• Diffusion Language Models: An Experimental Analysis
• MARLIN: De Novo Molecular Structure Elucidation from Tandem Mass Spectra without a Ground-Truth Formula
• Improved Large Language Diffusion Models
• Recursive Models for Long-Horizon Reasoning
• Think First, Diffuse Fast: Improving Diffusion Language Model Reasoning via Autoregressive Plan Conditioning
• Generalized Discrete Diffusion from Snapshots
• mRNAutilus: Multi-Objective-Guided Discrete Generation of mRNA with Optimized Therapeutic Properties
• AMix-2: Establishing Protein as a Native Modality in Large Language Models
• Ligand-Conditioned Discrete Diffusion for Protein Sequence-Structure Co-Design
• TD3B: Transition-Directed Discrete Diffusion for Allosteric Binder Generation
• Categorical Flow Maps
• Reviving Any-Subset Autoregressive Models with Principled Parallel Sampling and Speculative Decoding
• Conditional Monte Carlo Tree Diffusion for Designing Cell-Type-Specific and Biologically Faithful Regulatory DNA
• LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Model
• Diffusion Language Models for Speech Recognition
• Discrete Bridges for Mutual Information Estimation
• FOCUS: DLLMs Know How to Tame Their Compute Bound
• Minimal-Action Discrete Schrödinger Bridge Matching for Peptide Sequence Design
• LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models
• DELTA-TTS: Adapting Autoregressive Model into Diffusion Language Model for Text-to-Speech
• Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
• d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning
• d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillation
• α-Flow: A Unified Framework for Continuous-State Discrete Flow Matching Models
• Interaction-Required Suggestions for Control, Ownership, and Awareness in Human-AI Co-Writing
• Unifying Autoregressive and Diffusion-Based Sequence Generation
• RAG-ESM: Improving pretrained protein language models via sequence retrieval
• TabRep: Training Tabular Diffusion Models with a Simple and Effective Continuous Representation

Related

• Diffusion Beats Autoregressive in Data-Constrained Settings
• Why mask diffusion does not work
• DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models
• A Cheaper and Better Diffusion Language Model with Soft-Masked Noise
• Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling