Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models

2025/03/12 by Marianne Arriola, Aaron Gokaslan, Arriola, Marianne +13 · 17 voices · 113 citations
#cs.LG #cs.AI

paper · pdf · doi:10.48550/arxiv.2503.09573

Abstract

Diffusion language models offer unique benefits over autoregressive models due to their potential for parallelized generation and controllability, yet they lag in likelihood modeling and are limited to fixed-length generation. In this work, we introduce a class of block diffusion language models that interpolate between discrete denoising diffusion and autoregressive models. Block diffusion overcomes key limitations of both approaches by supporting flexible-length generation and improving inference efficiency with KV caching and parallel token sampling. We propose a recipe for building effective block diffusion models that includes an efficient training algorithm, estimators of gradient variance, and data-driven noise schedules to minimize the variance. Block diffusion sets a new state-of-the-art performance among diffusion models on language modeling benchmarks and enables generation of arbitrary-length sequences. We provide the code, along with the model weights and blog post on the project page: https://m-arriola.com/bd3lms

Citations

• Reasoning Beyond Words ? Exploring framework for hidden state reasoning
• Better & Faster Large Language Models via Multi-token Prediction
• Bayesian Flow Networks
• Stay on topic with Classifier-Free Guidance
• Affective Coherence Monitoring for Transformer-Based Language Models
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• Diffusion Models Beat GANs on Image Synthesis
• Attention Is All You Need

Cited by

• Understanding and Accelerating the Training of Masked Diffusion Language Models
• Context-weighted Discrete Flow Matching
• Ordered Action Tokens for Visuomotor Policy Learning
• LaViDa: A Large Diffusion Language Model for Multimodal Understanding
• Sparse-LaViDa: Sparse Multimodal Discrete Diffusion Language Models
• Multi-Mask Diffusion Language Models for Few-Step Generation
• UnMaskFork: Test-Time Scaling for Masked Diffusion via Deterministic Action Branching
• Mask-Aware Policy Gradients for Diffusion Language Models
• Polestar: Drift-Aware Cache Calibration and Token Commitment for Efficient Inference of Diffusion LLMs
• Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL
• Continuous Autoregressive Language Models
• LLaDA-MoE: A Sparse MoE Diffusion Language Model
• A Survey on Diffusion Language Models
• Large Language Diffusion Models
• Efficient Parallel Samplers for Recurrent-Depth Models and Their Connection to Diffusion Language Models
• WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference
• Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion Language Model Backbone
• DiRL: An Efficient Post-Training Framework for Diffusion Language Models
• Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusion
• Neuromorphic Diffusion Language Models: Addressing Compute and Memory Bottlenecks via Sparsity and Block Denoising
• TRE: Training-Free Hallucination Detection for Diffusion Language Models
• Beyond Block Boundaries: Multi-Block Editing for Diffusion Large Language Models
• Optimizing Decoding Paths in Masked Diffusion Models by Quantifying Uncertainty
• Fail Fast, Win Big: Rethinking the Drafting Strategy in Speculative Decoding via Diffusion LLMs
• PairFlow: Closed-Form Source-Target Coupling for Few-Step Generation in Discrete Flow Models
• Context-Aware Initialization for Reducing Generative Path Length in Diffusion Language Models
• DiffusionVL: Translating Any Autoregressive Models into Diffusion Vision Language Models
• Corrective Diffusion Language Models
• Fast and Accurate Causal Parallel Decoding using Jacobi Forcing
• Dual-objective Language Models: Training Efficiency Without Overfitting
• SDAR-VL: Stable and Efficient Block-wise Diffusion for Vision-Language Understanding
• Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speed
• ReFusion: A Diffusion Large Language Model with Parallel Autoregressive Decoding
• LLaDA2.0: Scaling Up Diffusion Language Models to 100B
• Learning Unmasking Policies for Diffusion Language Models
• From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs
• FASTer: Toward Efficient Autoregressive Vision Language Action Modeling via Neural Action Tokenization
• Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspective
• EDIT: Early Diffusion Inference Termination for dLLMs Based on Dynamics of Training Gradients
• BIFE: Better Interaction, Fewer Errors for Minute-Long Video Generation
• From Bits to Rounds: Parallel Decoding with Exploration for Diffusion Language Models
• PartDiffuser: Part-wise 3D Mesh Generation via Discrete Diffusion
• Bringing Stability to Diffusion: Decomposing and Reducing Variance of Training Masked Diffusion Models
• WavefrontDiffusion: Dynamic Decoding Schedule for Improved Reasoning
• DiffuGR: Generative Document Retrieval with Diffusion Language Models
• Diffusion Language Models are Super Data Learners
• Masked Diffusion Captioning for Visual Feature Learning
• Diffusion LLM with Native Variable Generation Lengths: Let [EOS] Lead the Way
• Variational Masked Diffusion Models
• Encoder-Decoder Diffusion Language Models for Efficient Training and Inference
• CANDI: Hybrid Discrete-Continuous Diffusion Models
• Parallel Sampling from Masked Diffusion Models via Conditional Independence Testing
• From Denoising to Refining: A Corrective Framework for Vision-Language Diffusion Model
• No Compute Left Behind: Rethinking Reasoning and Sampling with Masked Diffusion Models
• How Efficient Are Diffusion Language Models? A Critical Examination of Efficiency Evaluation Practices
• Planned Diffusion
• Soft-Masked Diffusion Language Models
• Planner and Executor: Collaboration between Discrete Diffusion And Autoregressive Models in Reasoning
• Attention Is All You Need for KV Cache in Diffusion LLMs
• On the Reasoning Abilities of Masked Diffusion Language Models
• UltraLLaDA: Scaling the Context Length to 128K for Diffusion Large Language Models
• SPG: Sandwiched Policy Gradient for Masked Diffusion Language Models
• Efficient Autoregressive Inference for Transformer Probabilistic Models
• Guided Star-Shaped Masked Diffusion
• Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimization
• Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimization
• Next Semantic Scale Prediction via Hierarchical Diffusion Language Models
• SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation
• Finish First, Perfect Later: Test-Time Token-Level Cross-Validation for Diffusion Large Language Models
• Test-Time Scaling in Diffusion LLMs via Hidden Semi-Autoregressive Experts
• ParallelBench: Understanding the Trade-offs of Parallel Decoding in Diffusion LLMs
• LaDiR: Latent Diffusion Enhances LLMs for Text Reasoning
• Demystifying MaskGIT Sampler and Beyond: Adaptive Order Selection in Masked Diffusion
• Self Speculative Decoding for Diffusion Large Language Models
• OneFlow: Concurrent Mixed-Modal and Interleaved Generation with Edit Flows
• Consolidating Reinforcement Learning for Multimodal Discrete Diffusion Models
• Free Draft-and-Verification: Toward Lossless Parallel Decoding for Diffusion Large Language Models
• dParallel: Learnable Parallel Decoding for dLLMs
• AdaBlock-dLLM: Semantic-Aware Diffusion LLM Inference via Adaptive Block Size
• Fast-dLLM v2: Efficient Block-Diffusion LLM
• RFG: Test-Time Scaling for Diffusion Large Language Model Reasoning with Reward-Free Guidance
• DiffuGuard: How Intrinsic Safety is Lost and Found in Diffusion Large Language Models
• Watermarking Diffusion Language Models
• TR2-D2: Tree Search Guided Trajectory-Aware Fine-Tuning for Discrete Diffusion
• Ultra-Fast Language Generation via Discrete Diffusion Divergence Instruct
• Alternatives To Next Token Prediction In Text Generation -- A Survey
• SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformer
• Learning to Parallel: Accelerating Diffusion Large Language Models via Learnable Parallel Decoding
• DiffuTester: Accelerating Unit Test Generation for Diffusion LLMs via Mining Structural Pattern
• Sequential Diffusion Language Models
• Taming Masked Diffusion Language Models via Consistency Trajectory Reinforcement Learning with Fewer Decoding Step
• Don't Settle Too Early: Self-Reflective Remasking for Diffusion Language Models
• Training Optimal Large Diffusion Language Models
• d²Cache: Accelerating Diffusion-Based LLMs via Dual Adaptive Caching
• Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action Spaces
• Soft-Di[M]O: Improving One-Step Discrete Image Generation with Soft Embeddings
• Discrete Guidance Matching: Exact Guidance for Discrete Flow Matching
• FS-DFM: Fast and Accurate Long Text Generation with Few-Step Diffusion Language Models
• DFlash: Block Diffusion for Flash Speculative Decoding
• Where-to-Unmask: Ground-Truth-Guided Unmasking Order Learning for Masked Diffusion Language Models
• Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning
• Inpainting-Guided Policy Optimization for Diffusion Large Language Models
• Set Block Decoding is a Language Model Inference Accelerator
• ParaThinker: Native Parallel Thinking as a New Paradigm to Scale LLM Test-time Compute
• Diffusion Language Models Know the Answer Before Decoding
• Blockwise SFT for Diffusion Language Models: Reconciling Bidirectional Attention and Autoregressive Decoding
• Dream 7B: Diffusion Large Language Models
• MDPO: Overcoming the Training-Inference Divide of Masked Diffusion Language Models
• MDNS: Masked Diffusion Neural Sampler via Stochastic Optimal Control
• Time Is a Feature: Exploiting Temporal Dynamics in Diffusion Language Models
• Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing
• Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference
• TreeDiff: AST-Guided Code Generation with Diffusion LLMs
• Wide-In, Narrow-Out: Revokable Decoding for Efficient and Effective DLLMs

Discussions

• Block Diffusion: Interpolating between autoregressive and diffusion models [hn, 156 points, 32 comments]
• A lot of really interesting HCI ideas here. Feels like there could be a whole paradigm hiding under this approach to interacting with text. I bet this would pair great with block diffusion. arxiv.or [bsky, 15 points, 0 comments]
• This paper looks pretty neat! Will be interesting to compare to SSD-LM which has a similar block autoregressive (with non-autoregressive chunks) approach arxiv.org/abs/2503.09573 [bsky, 9 points, 0 comments]
• Block Diffusion: Interpolating Between Autoregressive and Diffusion Models [bsky, 2 points, 0 comments]
• Stanford and Cornell just released "Block Diffusion" - an exciting new kid on the LLM block! It is quite exciting to see that LLMs may finally have reached their paralle-inference moment: Paper: arxi [bsky, 2 points, 0 comments]
• 🤔 Interesting work presenting block diffusion as a promising technique that bridges autoregressive and diffusion methodologies, bringing about advancements in flexibility, efficiency, and sequence ge [bsky, 1 points, 0 comments]
• Block Diffusion: Interpolating Between Autoregressive and Diffusion Models https://arxiv.org/abs/2503.09573 [bsky, 0 points, 0 comments]
• Block Diffusion: Interpolating between autoregressive and diffusion models https://arxiv.org/abs/2503.09573 [comments] [132 points] [bsky, 0 points, 0 comments]
• Paper: arxiv.org/abs/2503.09573 Code, model weights: m-arriola.com/bd3lms/ [bsky, 0 points, 0 comments]
• ⚡ Hackernews Top story: Block Diffusion: Interpolating Between Autoregressive and Diffusion Models [bsky, 0 points, 0 comments]
• https://bsky.app/profile/news.ycombinator.com.web.brid.gy/post/3lkdyl2pmjxt2 [bsky, 0 points, 0 comments]
• Block Diffusion: Interpolating Between Autoregressive and Diffusion Models (arxiv.org) Main Link | Discussion [bsky, 0 points, 0 comments]
• Block Diffusion: Interpolating Between Autoregressive and Diffusion Models view on hacker news [bsky, 0 points, 0 comments]
• Block Diffusion: Interpolating Between Autoregressive and Diffusion Models https://arxiv.org/abs/2503.09573 (https://news.ycombinator.com/item?id=43363247) [bsky, 0 points, 0 comments]
• Block Diffusion: Interpolating Between Autoregressive and Diffusion Models https://arxiv.org/abs/2503.09573 (https://news.ycombinator.com/item?id=43363247) [bsky, 0 points, 0 comments]
• Block Diffusion: Interpolating Between Autoregressive and Diffusion Models #HackerNews https://arxiv.org/abs/2503.09573 [bsky, 0 points, 0 comments]
• Block Diffusion: Interpolating between autoregressive and diffusion models https://arxiv.org/abs/2503.09573 https://news.ycombinator.com/item?id=43363247 [bsky, 0 points, 0 comments]

Related

• Likelihood-Based Diffusion Language Models
• FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusion
• Diffusion Beats Autoregressive in Data-Constrained Settings
• Encoder-Decoder Diffusion Language Models for Efficient Training and Inference
• Rethinking the Generation Order of Block Diffusion Language Models