Simple and Effective Masked Diffusion Language Models

2024/06/11 by Subham Sekhar Sahoo, Marianne Arriola, Sahoo, Subham Sekhar +13 · 167 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2406.07524

openalex publication_date 2024/06/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

While diffusion models excel at generating high-quality images, prior work reports a significant performance gap between diffusion and autoregressive (AR) methods in language modeling. In this work, we show that simple masked discrete diffusion is more performant than previously thought. We apply an effective training recipe that improves the performance of masked diffusion models and derive a simplified, Rao-Blackwellized objective that results in additional improvements. Our objective has a simple form -- it is a mixture of classical masked language modeling losses -- and can be used to train encoder-only language models that admit efficient samplers, including ones that can generate arbitrary lengths of text semi-autoregressively like a traditional language model. On language modeling benchmarks, a range of masked diffusion models trained with modern engineering practices achieves a new state-of-the-art among diffusion models, and approaches AR perplexity. We provide the code, along with a blog post and video tutorial on the project page: https://s-sahoo.com/mdlm

Cited by

• UNIFUSION: Adapting Autoregressive Language Models into Discrete Diffusion under a Unified Reverse-Rate Objective
• dUltra: Ultra-Fast Diffusion Language Models via Reinforcement Learning
• On the Role of Discreteness in Diffusion LLMs
• Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion Language Model Backbone
• Where Quality Breaks in Compressed Short-Text Generation: Staged Bottleneck Localization
• PreDiff-LM: Pretrained Discrete Masked Diffusion Language Modeling with Hybrid Attention
• TRE: Training-Free Hallucination Detection for Diffusion Language Models
• FRIGID: Scaling Diffusion-Based Molecular Generation from Mass Spectra at Training and Inference Time
• Optimizing Decoding Paths in Masked Diffusion Models by Quantifying Uncertainty
• PairFlow: Closed-Form Source-Target Coupling for Few-Step Generation in Discrete Flow Models
• Corrective Diffusion Language Models
• Dual-objective Language Models: Training Efficiency Without Overfitting
• ViewMask-1-to-3: Multi-View Consistent Image Generation via Multimodal Discrete Diffusion Models
• SDAR-VL: Stable and Efficient Block-wise Diffusion for Vision-Language Understanding
• Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speed
• Scaling Behavior of Discrete Diffusion Language Models
• Guided Transfer Learning for Discrete Diffusion Models
• Learning Unmasking Policies for Diffusion Language Models
• From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs
• Generative Recursive Reasoning
• Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspective
• dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoning
• MDiff4STR: Mask Diffusion Model for Scene Text Recognition
• MM-ACT: Learn from Multimodal Parallel Generation to Act
• Masked Diffusion for Generative Recommendation
• Masks Can Be Distracting: On Context Comprehension in Diffusion Language Models
• Beyond Confidence: Adaptive and Coherent Decoding for Diffusion Language Models
• Which Layer Causes Distribution Deviation? Entropy-Guided Adaptive Pruning for Diffusion and Flow Models
• From Bits to Rounds: Parallel Decoding with Exploration for Diffusion Language Models
• Orchestrating Dual-Boundaries: An Arithmetic Intensity Inspired Acceleration Framework for Diffusion Language Models
• Masked Diffusion Models are Secretly Learned-Order Autoregressive Models
• Demystifying Diffusion Objectives: Reweighted Losses are Better Variational Bounds
• Bringing Stability to Diffusion: Decomposing and Reducing Variance of Training Masked Diffusion Models
• WavefrontDiffusion: Dynamic Decoding Schedule for Improved Reasoning
• MDG: Masked Denoising Generation for Multi-Agent Behavior Modeling in Traffic Environments
• Branching Flows: Discrete, Continuous, and Manifold Flow Matching with Splits and Deletions
• KLASS: KL-Guided Fast Inference in Masked Diffusion Models
• Optimal Inference Schedules for Masked Diffusion Models
• Diffusion Language Models are Super Data Learners
• Lookahead Unmasking Elicits Accurate Decoding in Diffusion Language Models
• Effective Test-Time Scaling of Discrete Diffusion through Iterative Refinement
• SpecDiff-2: Scaling Diffusion Drafter Alignment For Faster Speculative Decoding
• Simplex-to-Euclidean Bijections for Categorical Flow Matching
• Masked Diffusion Captioning for Visual Feature Learning
• Don't Let It Fade: Preserving Edits in Diffusion Language Models via Token Timestep Allocation
• Error Bounds and Optimal Schedules for Masked Diffusions with Factorized Approximations
• From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models
• How to make the most of your masked language model for protein engineering
• Information-Theoretic Discrete Diffusion
• Variational Masked Diffusion Models
• Sampling from Energy distributions with Target Concrete Score Identity
• Simple Denoising Diffusion Language Models
• Encoder-Decoder Diffusion Language Models for Efficient Training and Inference
• CANDI: Hybrid Discrete-Continuous Diffusion Models
• Aligning Diffusion Language Models via Unpaired Preference Optimization
• Parallel Sampling from Masked Diffusion Models via Conditional Independence Testing
• Efficient semantic uncertainty quantification in language models via diversity-steered sampling
• From Masks to Worlds: A Hitchhiker's Guide to World Models
• Loopholing Discrete Diffusion: Deterministic Bypass of the Sampling Wall
• From Denoising to Refining: A Corrective Framework for Vision-Language Diffusion Model
• DiffGRM: Diffusion-based Generative Recommendation Model
• Latent-Augmented Discrete Diffusion Models
• Planned Diffusion
• Soft-Masked Diffusion Language Models
• Planner and Executor: Collaboration between Discrete Diffusion And Autoregressive Models in Reasoning
• Attention Is All You Need for KV Cache in Diffusion LLMs
• Inference-Time Search using Side Information for Diffusion-based Image Reconstruction
• Test-Time Anchoring for Discrete Diffusion Posterior Sampling
• UltraLLaDA: Scaling the Context Length to 128K for Diffusion Large Language Models
• On the Quantization Robustness of Diffusion Language Models in Coding Benchmarks
• SPG: Sandwiched Policy Gradient for Masked Diffusion Language Models
• Diffusion-Inspired Masked Fine-Tuning for Knowledge Injection in Autoregressive LLMs
• Guided Star-Shaped Masked Diffusion
• Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimization
• Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimization
• Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistency
• Next Semantic Scale Prediction via Hierarchical Diffusion Language Models
• DiFFPO: Training Diffusion LLMs to Reason Fast and Furious via Reinforcement Learning
• Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understanding
• SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation
• Discrete Diffusion Models with MLLMs for Unified Medical Multimodal Generation
• Improving Discrete Diffusion Unmasking Policies Beyond Explicit Reference Policies
• Membership Inference Attacks on Tokenizers of Large Language Models
• Fine-Tuning Masked Diffusion for Provable Self-Correction
• Finish First, Perfect Later: Test-Time Token-Level Cross-Validation for Diffusion Large Language Models
• ParallelBench: Understanding the Trade-offs of Parallel Decoding in Diffusion LLMs
• LaDiR: Latent Diffusion Enhances LLMs for Text Reasoning
• Demystifying MaskGIT Sampler and Beyond: Adaptive Order Selection in Masked Diffusion
• Drax: Speech Recognition with Discrete Flow Matching
• Rainbow Padding: Mitigating Early Termination in Instruction-Tuned Diffusion LLMs
• Self-Speculative Masked Diffusions
• OneFlow: Concurrent Mixed-Modal and Interleaved Generation with Edit Flows
• DMark: Order-Agnostic Watermarking for Diffusion Large Language Models
• Consolidating Reinforcement Learning for Multimodal Discrete Diffusion Models
• Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasoner
• Toward Safer Diffusion Language Models: Discovery and Mitigation of Priming Vulnerability
• Diffusion Alignment as Variational Expectation-Maximization
• Continuously Augmented Discrete Diffusion model for Categorical Generative Modeling
• AReUReDi: Annealed Rectified Updates for Refining Discrete Flows with Multi-Objective Guidance
• Free Draft-and-Verification: Toward Lossless Parallel Decoding for Diffusion Large Language Models
• dParallel: Learnable Parallel Decoding for dLLMs
• Refine Drugs, Don't Complete Them: Uniform-Source Discrete Flows for Fragment-Based Drug Discovery
• dVLA: Diffusion Vision-Language-Action Model with Multimodal Chain-of-Thought
• TraceDet: Hallucination Detection from the Decoding Trace of Diffusion Large Language Models
• RFG: Test-Time Scaling for Diffusion Large Language Model Reasoning with Reward-Free Guidance
• Watermarking Diffusion Language Models
• TR2-D2: Tree Search Guided Trajectory-Aware Fine-Tuning for Discrete Diffusion
• Deep Thinking by Markov Chain of Continuous Thoughts
• Ultra-Fast Language Generation via Discrete Diffusion Divergence Instruct
• Learning to Parallel: Accelerating Diffusion Large Language Models via Learnable Parallel Decoding
• DiffuTester: Accelerating Unit Test Generation for Diffusion LLMs via Mining Structural Pattern
• Taming Masked Diffusion Language Models via Consistency Trajectory Reinforcement Learning with Fewer Decoding Step
• Training Optimal Large Diffusion Language Models
• Electric Currents for Discrete Data Generation
• RIV: Recursive Introspection Mask Diffusion Vision Language Model
• Planner Aware Path Learning in Diffusion Language Models Training
• Entering the Era of Discrete Diffusion Models: A Benchmark for Schrödinger Bridges and Entropic Optimal Transport
• Tree Reward-Aligned Search for TReASURe in Masked Diffusion Language Models
• d²Cache: Accelerating Diffusion-Based LLMs via Dual Adaptive Caching
• Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action Spaces
• Soft-Di[M]O: Improving One-Step Discrete Image Generation with Soft Embeddings
• Scale-Wise VAR is Secretly Discrete Diffusion
• Decoding quantum low density parity check codes with diffusion
• Discrete Guidance Matching: Exact Guidance for Discrete Flow Matching
• Error Analysis of Discrete Flow with Generator Matching
• Enabling Approximate Joint Sampling in Diffusion LMs
• Deterministic Discrete Denoising
• Understanding and Enhancing Mask-Based Pretraining towards Universal Representations
• d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimation
• FS-DFM: Fast and Accurate Long Text Generation with Few-Step Diffusion Language Models
• From Text to Talk: Audio-Language Model Needs Non-Autoregressive Joint Training
• Discrete Diffusion for Generative Modeling of Text-Aligned Speech Tokens
• Learnable Sampler Distillation for Discrete Diffusion Models
• Where and When to Commit: Candidate-Aware Decoding for Diffusion Language Models
• Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation
• ELF: Embedded Language Flows
• Dream-Cubed: Controllable Generative Modeling in Minecraft by Training on Billions of Cubes
• Where-to-Unmask: Ground-Truth-Guided Unmasking Order Learning for Masked Diffusion Language Models
• Audio-Conditioned Diffusion LLMs for ASR and Deliberation Processing
• Monte Carlo Tree Diffusion with Multiple Experts for Protein Design
• Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning
• Masked Diffusion Models as Energy Minimization
• DSFT: Inspiring Diffusion Large Language Models to Comprehend Mathematical and Logical Patterns
• ShortListing Model: A Streamlined SimplexDiffusion for Discrete Variable Generation
• Beyond Autoregression: An Empirical Study of Diffusion Large Language Models for Code Generation
• Inpainting-Guided Policy Optimization for Diffusion Large Language Models
• LLaDA-VLA: Vision Language Diffusion Action Models
• Masked Diffusion Language Models with Frequency-Informed Training
• Dream-Coder 7B: An Open Diffusion Language Model for Code
• Any-Order Flexible Length Masked Diffusion
• Diffusion Language Models Know the Answer Before Decoding
• Blockwise SFT for Diffusion Language Models: Reconciling Bidirectional Attention and Autoregressive Decoding
• Scaffold Diffusion: Sparse Multi-Category Voxel Structure Generation with Discrete Diffusion
• Dream 7B: Diffusion Large Language Models
• Gener <i>anno</i> : A Genomic Foundation Model for Metagenomic Annotation
• ToolACE-MT: Non-Autoregressive Generation for Agentic Multi-Turn Interaction
• MDPO: Overcoming the Training-Inference Divide of Masked Diffusion Language Models
• Where to Start Alignment? Diffusion Large Language Model May Demand a Distinct Position
• Improving Text Style Transfer using Masked Diffusion Language Models with Inference-time Scaling
• Thinking Inside the Mask: In-Place Prompting in Diffusion LLMs
• MDNS: Masked Diffusion Neural Sampler via Stochastic Optimal Control
• Time Is a Feature: Exploiting Temporal Dynamics in Diffusion Language Models
• The Cosine Schedule is Fisher-Rao-Optimal for Masked Discrete Diffusion Models
• One Model for All: Unified Try-On and Try-Off in Any Pose via LLM-Inspired Bidirectional Tweedie Diffusion
• Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference
• LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understanding
• TreeDiff: AST-Guided Code Generation with Diffusion LLMs

Related

• Diffusion Beats Autoregressive in Data-Constrained Settings
• Why mask diffusion does not work
• DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models
• A Cheaper and Better Diffusion Language Model with Soft-Masked Noise
• Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling