DLM-One: Diffusion Language Models for One-Step Sequence Generation

2025/05/30 by Tianqi Chen, Shujian Zhang, Chen, Tianqi +3 · 4 citations
Computer Science · #Benchmark (surveying) #Computation and Language (cs.CL) #Diffusion #Embedding #FOS: Computer and information sciences #Generality #Inference #Language model #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Model-Driven Software Engineering Techniques #Natural Language Processing Techniques #Sequence (biology) #Speech and dialogue systems #Speedup

paper · pdf · doi:10.48550/arxiv.2506.00290

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/05/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

This paper introduces DLM-One, a score-distillation-based framework for one-step sequence generation with continuous diffusion language models (DLMs). DLM-One eliminates the need for iterative refinement by aligning the scores of a student model's outputs in the continuous token embedding space with the score function of a pretrained teacher DLM. We investigate whether DLM-One can achieve substantial gains in sampling efficiency for language modeling. Through comprehensive experiments on DiffuSeq -- a representative continuous DLM -- we show that DLM-One achieves up to ~500x speedup in inference time while maintaining competitive performance on benchmark text generation tasks used to evaluate the teacher models. We further analyze the method's empirical behavior across multiple datasets, providing initial insights into its generality and practical applicability. Our findings position one-step diffusion as a promising direction for efficient, high-quality language generation and broader adoption of continuous diffusion models operating in embedding space for natural language processing.

Citations

• Few-Step Diffusion via Score identity Distillation
• Large Language Diffusion Models
• Scaling up Masked Diffusion Models on Text
• Scaling Diffusion Language Models via Adaptation from Autoregressive Models
• Adversarial Score identity Distillation: Rapidly Surpassing the Teacher in One Step
• Plug-and-Play Controllable Generation for Discrete Masked Models
• Discrete Copula Diffusion
• Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling
• The Llama 3 Herd of Models
• Consistency Models Made Easy
• Simple and Effective Masked Diffusion Language Models
• Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data
• Simplified and Generalized Masked Diffusion for Discrete Data
• Improved Distribution Matching Distillation for Fast Image Synthesis
• A Comprehensive Survey of Accelerated Generation Techniques in Large Language Models
• Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation
• The pitfalls of next-token prediction
• TEncDM: Understanding the Properties of the Diffusion Model in the Space of Language Model Encodings
• One-step Diffusion with Distribution Matching Distillation
• Adversarial Diffusion Distillation
• Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution
• Improved Techniques for Training Consistency Models
• The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"
• Challenges and Applications of Large Language Models
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model
• Likelihood-Based Diffusion Language Models
• Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models
• ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation
• AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation
• TESS: Text-to-Text Self-Conditioned Simplex Diffusion
• Diffusion-NAT: Self-Prompting Discrete Diffusion for Non-Autoregressive Text Generation
• GPT-4 Technical Report
• Diffusion Models for Non-autoregressive Text Generation: A Survey
• Consistency Models
• DINOISER: Diffused Conditional Sequence Learning by Manipulating Noises
• A Reparameterized Discrete Diffusion Model for Text Generation
• Q-Diffusion: Quantizing Diffusion Models
• Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoise
• SeqDiffuSeq: Text Diffusion with Encoder-Decoder Transformers
• Empowering Diffusion Models on the Embedding Space for Text Generation
• Latent Diffusion for Language Generation
• Continuous diffusion for categorical data
• Self-conditioned Embedding Diffusion for Text Generation
• Concrete Score Matching: Generalized Score Matching for Discrete Data
• SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control
• DiffusER: Discrete Diffusion via Edit-based Reconstruction
• DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models
• DreamFusion: Text-to-3D using 2D Diffusion
• Classifier-Free Diffusion Guidance
• Diffusion-GAN: Training GANs with Diffusion
• DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps
• Diffusion-LM Improves Controllable Text Generation
• PaLM: Scaling Language Modeling with Pathways
• Pseudo Numerical Methods for Diffusion Models on Manifolds
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• Global Context with Discrete Diffusion in Vector Quantised Modelling for Image Generation
• Vector Quantized Diffusion Model for Text-to-Image Synthesis
• Structured Denoising Diffusion Models in Discrete State-Spaces
• Diffusion Models Beat GANs on Image Synthesis
• Improved Denoising Diffusion Probabilistic Models
• Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions
• Taming Transformers for High-Resolution Image Synthesis
• Score-Based Generative Modeling through Stochastic Differential Equations
• Limitations of Autoregressive Models and Their Alternatives
• Denoising Diffusion Implicit Models
• Denoising Diffusion Probabilistic Models
• Language Models are Few-Shot Learners
• Neural CRF Model for Sentence Alignment in Text Simplification
• Plug and Play Language Models: A Simple Approach to Controlled Text Generation
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• BERTScore: Evaluating Text Generation with BERT
• Texygen: A Benchmarking Platform for Text Generation Models
• Neural Discrete Representation Learning
• Quasar: Datasets for Question Answering by Search and Reading
• Attention Is All You Need
• Tweedie’s Formula and Selection Bias
• Guided Score identity Distillation for Data-Free One-Step Text-to-Image Generation

Cited by

• Fast-Decoding Diffusion Language Models via Progress-Aware Confidence Schedules
• How Efficient Are Diffusion Language Models? A Critical Examination of Efficiency Evaluation Practices
• Beyond Next-Token Prediction: A Performance Characterization of Diffusion versus Autoregressive Language Models
• FS-DFM: Fast and Accurate Long Text Generation with Few-Step Diffusion Language Models

Related

• Diffusion Language Models: An Experimental Analysis
• FS-DFM: Fast and Accurate Long Text Generation with Few-Step Diffusion Language Models
• DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models
• Diffusion Language Models Generation Can Be Halted Early
• A Survey on Diffusion Language Models