From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs

2025/12/07 by Tian, Yuchuan, Liang, Yuchen, Sun, Jiacheng +11 · 4 citations
Computer Science · Medicine · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Healthcare and Education #Computation and Language (cs.CL) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Topic Modeling

paper · doi:10.48550/arxiv.2512.06776

openalex publication_date 2025/12/07 · openalex created_date 2025/12/10 · openalex updated_date 2026/07/28

Abstract

Diffusion Language Models (DLMs) enable fast generation, yet training large DLMs from scratch is costly. As a practical shortcut, adapting off-the-shelf Auto-Regressive (AR) model weights into a DLM could quickly equip the DLM with strong long-context generation capabilies. Prior "adaptation" attempts either modify logits or randomly grow attention masks to Full-Sequence diffusion, or simply transplant AR weights into a Block-Diffusion recipe, leaving two key questions unaddressed: where is the final destination of adaptation, and how to adapt better? For manifold benefits, we reframe the whole AR-to-DLM adaptation under the Block-Diffusion paradigm, transitioning from block size 1 to the final Block-Diffusion state. Concretely, the principled pathway of adaptation is designed as follows: we keep a context-causal path where causal attention is kept in the prefix, an efficient parallel adaptation procedure where an AR guidance is maintained, and gradual increment of the generation block size for a smoother transition. Built on these components, the adaptation is proved competitive on various models at different scales. With better adaptation, we propose NBDiff-7B that could inherit the long-context modeling and reasoning capabilities, and achieve state-of-the-art performance among the 7B-class DLMs. Codes: https://github.com/YuchuanTian/NBDiff.

Citations

• SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation
• d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimation
• Thinking Inside the Mask: In-Place Prompting in Diffusion LLMs
• Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing
• wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models
• DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation
• Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding
• LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning
• dKV-Cache: The Cache for Diffusion Language Models
• dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Caching
• Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models
• Remasking Discrete Diffusion Models with Inference-Time Scaling
• Large Language Diffusion Models
• Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions
• Scaling Diffusion Language Models via Adaptation from Autoregressive Models
• Simple and Effective Masked Diffusion Language Models
• Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data
• MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
• Diffusion of Thoughts: Chain-of-Thought Reasoning in Diffusion Language Models
• Instruction-Following Evaluation for Large Language Models
• Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
• Diffusion-LM Improves Controllable Text Generation
• Training Verifiers to Solve Math Word Problems
• Program Synthesis with Large Language Models
• Structured Denoising Diffusion Models in Discrete State-Spaces
• Evaluating Large Language Models Trained on Code
• Measuring Massive Multitask Language Understanding

Cited by

• WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference
• Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusion
• Rethinking the Generation Order of Block Diffusion Language Models
• TEAM: Temporal-Spatial Consistency Guided Expert Activation for MoE Diffusion Language Model Acceleration

Related

• WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference
• Fast-dLLM v2: Efficient Block-Diffusion LLM
• Sequential Diffusion Language Models
• Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speed
• Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning