Large Language Diffusion Models

2025/02/14 by Shen Nie, Fengqi Zhu, Nie, Shen +17 · 19 voices · 353 citations
Computer Science · #Computer science #Diffusion #Natural Language Processing Techniques #Physics #Statistical physics #Thermodynamics #Topic Modeling #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2502.09992

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/02/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

The capabilities of large language models (LLMs) are widely regarded as relying on autoregressive models (ARMs). We challenge this notion by introducing LLaDA, a diffusion model trained from scratch under the pre-training and supervised fine-tuning (SFT) paradigm. LLaDA employs a forward data masking process and a reverse generation process, parameterized by a Transformer to predict masked tokens. It provides a principled generative approach for probabilistic inference by optimizing a likelihood lower bound. Across extensive benchmarks on general tasks, math, code, and so on, LLaDA demonstrates strong scalability and performs comparably to our self-constructed ARM baselines. Remarkably, LLaDA 8B is competitive with strong LLMs like LLaMA3 8B in in-context learning and, after SFT, exhibits impressive instruction-following abilities in case studies such as multi-turn dialogue. Moreover, LLaDA addresses the reversal curse, surpassing GPT-4o in a reversal poem completion task. Our findings show the promise of diffusion models for language modeling at scale and challenge the common assumption that core LLM capabilities discussed above inherently depend on ARMs. Project page and codes: https://ml-gsai.github.io/LLaDA-demo/.

Citations

• Mercury: Ultra-Fast Language Models Based on Diffusion
• Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models
• Magicoder: Empowering Code Generation with OSS-Instruct
• Mistral 7B
• The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"
• Language Modeling Is Compression
• Bayesian Flow Networks
• Generative Agents: Interactive Simulacra of Human Behavior
• Training Compute-Optimal Large Language Models
• Training language models to follow instructions with human feedback
• Program Synthesis with Large Language Models
• Diffusion Models Beat GANs on Image Synthesis
• Language Models are Few-Shot Learners
• Attention Is All You Need

Cited by

• Context-weighted Discrete Flow Matching
• LaViDa: A Large Diffusion Language Model for Multimodal Understanding
• Sparse-LaViDa: Sparse Multimodal Discrete Diffusion Language Models
• Lavida-O: Elastic Large Masked Diffusion Models for Unified Multimodal Understanding and Generation
• Dimension-Free Convergence of Discrete Diffusion Models: Adjoint Equations Induce the Right Space
• Multi-Mask Diffusion Language Models for Few-Step Generation
• Tensor-Train Joint Modeling for Few-Step Discrete Diffusion
• One-step lowest-variance selection in a Gaussian random-field model motivated by masked diffusion: Total correlation and a square root collision threshold
• ST-Veto: Spatio-Temporal Token Veto for Diffusion MLLMs via Taylor Prediction and Visual Grounding
• UnMaskFork: Test-Time Scaling for Masked Diffusion via Deterministic Action Branching
• Breaking the Factorization Barrier in Diffusion Language Models
• A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Models
• Mask-Aware Policy Gradients for Diffusion Language Models
• UNIFUSION: Adapting Autoregressive Language Models into Discrete Diffusion under a Unified Reverse-Rate Objective
• DC-Leap: Training-Free Acceleration of dLLMs via Draft-Guided Contiguous Leaping Decoding
• JUMP: Single-Pass Membership Inference on Fine-Tuned Diffusion Language Models
• Polestar: Drift-Aware Cache Calibration and Token Commitment for Efficient Inference of Diffusion LLMs
• Token Time Continuous Diffusion for Language Modeling
• Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL
• Speculative Speculative Decoding
• Mind captioning: Evolving descriptive text of mental content from human brain activity
• LLaDA-MoE: A Sparse MoE Diffusion Language Model
• A Survey on Diffusion Language Models
• Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential
• Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding
• Steering Generative Models with Experimental Data for Protein Fitness Optimization
• Neurosymbolic Diffusion Models
• Idiosyncrasies in Large Language Models
• dUltra: Ultra-Fast Diffusion Language Models via Reinforcement Learning
• Plug-and-Play Fidelity Optimization for Diffusion Transformer Acceleration via Cumulative Error Minimization
• WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference
• Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion Language Model Backbone
• Self-Evaluation Unlocks Any-Step Text-to-Image Generation
• Co-GRPO: Co-Optimized Group Relative Policy Optimization for Masked Diffusion Model
• DiRL: An Efficient Post-Training Framework for Diffusion Language Models
• Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusion
• PreDiff-LM: Pretrained Discrete Masked Diffusion Language Modeling with Hybrid Attention
• TRE: Training-Free Hallucination Detection for Diffusion Language Models
• Beyond Block Boundaries: Multi-Block Editing for Diffusion Large Language Models
• CaRE Compute-aware Remasking Evaluation Protocol for Masked Diffusion Language Models
• Optimizing Decoding Paths in Masked Diffusion Models by Quantifying Uncertainty
• ReaSeq: Unleashing World Knowledge via Reasoning for Sequential Modeling
• Fail Fast, Win Big: Rethinking the Drafting Strategy in Speculative Decoding via Diffusion LLMs
• PairFlow: Closed-Form Source-Target Coupling for Few-Step Generation in Discrete Flow Models
• dMLLM-TTS: Self-Verified and Efficient Test-Time Scaling for Diffusion Multi-Modal Large Language Models
• DiffusionVL: Translating Any Autoregressive Models into Diffusion Vision Language Models
• DEER: Draft with Diffusion, Verify with Autoregressive Models
• HD-Prot: A Protein Language Model for Joint Sequence-Structure Modeling with Continuous Structure Tokens
• Fast and Accurate Causal Parallel Decoding using Jacobi Forcing
• Dual-objective Language Models: Training Efficiency Without Overfitting
• ViewMask-1-to-3: Multi-View Consistent Image Generation via Multimodal Discrete Diffusion Models
• Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speed
• ReFusion: A Diffusion Large Language Model with Parallel Autoregressive Decoding
• LINA: Learning INterventions Adaptively for Physical Alignment and Generalization in Diffusion Models
• MineTheGap: Automatic Mining of Biases in Text-to-Image Models
• Scaling Behavior of Discrete Diffusion Language Models
• d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Models
• Learning Unmasking Policies for Diffusion Language Models
• Beyond Real: Imaginary Extension of Rotary Position Embeddings for Long-Context LLMs
• From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs
• WAM-Diff: A Masked Diffusion VLA Framework with MoE and Online Reinforcement Learning for Autonomous Driving
• WAM-Flow: Parallel Coarse-to-Fine Motion Planning via Discrete Flow Matching for Autonomous Driving
• Fast SceneScript: Accurate and Efficient Structured Language Model via Multi-Token Prediction
• Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspective
• dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoning
• Self-Paced and Self-Corrective Masked Prediction for Movie Trailer Generation
• Decoding Large Language Diffusion Models with Foreseeing Movement
• PretrainZero: Reinforcement Active Pretraining
• Fast-Decoding Diffusion Language Models via Progress-Aware Confidence Schedules
• Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learning
• TUNA: Taming Unified Visual Representations for Native Unified Multimodal Models
• Syndrome-Flow Consistency Model Achieves One-step Denoising Error Correction Codes
• Accelerating Inference of Masked Image Generators via Reinforcement Learning
• MM-ACT: Learn from Multimodal Parallel Generation to Act
• EDIT: Early Diffusion Inference Termination for dLLMs Based on Dynamics of Training Gradients
• Markovian Scale Prediction: A New Era of Visual Autoregressive Generation
• Closed-Loop Transformers: Autoregressive Modeling as Iterative Latent Equilibrium
• E₀: Enhancing Generalization and Fine-Grained Control in VLA Models via Continuized Discrete Diffusion
• Masks Can Be Distracting: On Context Comprehension in Diffusion Language Models
• Beyond Confidence: Adaptive and Coherent Decoding for Diffusion Language Models
• From Bits to Rounds: Parallel Decoding with Exploration for Diffusion Language Models
• CDLM: Consistency Diffusion Language Models For Faster Sampling
• Fluid Intelligence: A Forward Look on AI Foundation Models in Computational Fluid Dynamics
• Orchestrating Dual-Boundaries: An Arithmetic Intensity Inspired Acceleration Framework for Diffusion Language Models
• FlowSteer: Guiding Few-Step Image Synthesis with Authentic Trajectories
• PartDiffuser: Part-wise 3D Mesh Generation via Discrete Diffusion
• DELTA: Language Diffusion-based EEG-to-Text Architecture
• Demystifying Diffusion Objectives: Reweighted Losses are Better Variational Bounds
• Bringing Stability to Diffusion: Decomposing and Reducing Variance of Training Masked Diffusion Models
• WavefrontDiffusion: Dynamic Decoding Schedule for Improved Reasoning
• Reasoning in Diffusion Large Language Models is Concentrated in Dynamic Confusion Zones
• A Comprehensive Study on Visual Token Redundancy for Discrete Diffusion-based Multimodal Large Language Models
• AsyncVLA: Asynchronous Flow Matching for Vision-Language-Action Models
• MMaDA-Parallel: Multimodal Large Diffusion Language Models for Thinking-Aware Editing and Generation
• KLASS: KL-Guided Fast Inference in Masked Diffusion Models
• Optimal Inference Schedules for Masked Diffusion Models
• Forget BIT, It is All about TOKEN: Towards Semantic Information Theory for LLMs
• Diffusion Language Models are Super Data Learners
• Lookahead Unmasking Elicits Accurate Decoding in Diffusion Language Models
• Effective Test-Time Scaling of Discrete Diffusion through Iterative Refinement
• Watermarking Discrete Diffusion Language Models
• The Era of Agentic Organization: Learning to Organize with Language Models
• Don't Let It Fade: Preserving Edits in Diffusion Language Models via Token Timestep Allocation
• PairUni: Pairwise Training for Unified Multimodal Language Models
• From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models
• Speech-XL: Towards Long-Form Speech Understanding in Large Speech Language Models
• Information-Theoretic Discrete Diffusion
• Variational Masked Diffusion Models
• Simple Denoising Diffusion Language Models
• Encoder-Decoder Diffusion Language Models for Efficient Training and Inference
• Aligning Diffusion Language Models via Unpaired Preference Optimization
• Parallel Sampling from Masked Diffusion Models via Conditional Independence Testing
• Efficient semantic uncertainty quantification in language models via diversity-steered sampling
• Noise is All You Need: Solving Linear Inverse Problems by Noise Combination Sampling with Diffusion Models
• From Masks to Worlds: A Hitchhiker's Guide to World Models
• A Survey on Cache Methods in Diffusion Models: Toward Efficient Multi-Modal Generation
• Loopholing Discrete Diffusion: Deterministic Bypass of the Sampling Wall
• From Denoising to Refining: A Corrective Framework for Vision-Language Diffusion Model
• No Compute Left Behind: Rethinking Reasoning and Sampling with Masked Diffusion Models
• How Efficient Are Diffusion Language Models? A Critical Examination of Efficiency Evaluation Practices
• Latent-Augmented Discrete Diffusion Models
• Planned Diffusion
• Soft-Masked Diffusion Language Models
• Saber: An Efficient Sampling with Adaptive Acceleration and Backtracking Enhanced Remasking for Diffusion Language Model
• Planner and Executor: Collaboration between Discrete Diffusion And Autoregressive Models in Reasoning
• Attention Is All You Need for KV Cache in Diffusion LLMs
• On the Reasoning Abilities of Masked Diffusion Language Models
• NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matching
• DiSTAR: Diffusion over a Scalable Token Autoregressive Representation for Speech Generation
• Test-Time Anchoring for Discrete Diffusion Posterior Sampling
• Latent Refinement Decoding: Enhancing Diffusion-Based Language Models by Refining Belief States
• UltraLLaDA: Scaling the Context Length to 128K for Diffusion Large Language Models
• On the Quantization Robustness of Diffusion Language Models in Coding Benchmarks
• Every Step Counts: Decoding Trajectories as Authorship Fingerprints of dLLMs
• Beyond Surface Reasoning: Unveiling the True Long Chain-of-Thought Capacity of Diffusion Large Language Models
• SPG: Sandwiched Policy Gradient for Masked Diffusion Language Models
• Diffusion-Inspired Masked Fine-Tuning for Knowledge Injection in Autoregressive LLMs
• dInfer: An Efficient Inference Framework for Diffusion Language Models
• Guided Star-Shaped Masked Diffusion
• Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimization
• Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimization
• Accelerating Diffusion LLM Inference via Local Determinism Propagation
• DiFFPO: Training Diffusion LLMs to Reason Fast and Furious via Reinforcement Learning
• OBS-Diff: Accurate Pruning For Diffusion Models in One-Shot
• AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMs
• Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understanding
• SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation
• Discrete Diffusion Models with MLLMs for Unified Medical Multimodal Generation
• Improving Discrete Diffusion Unmasking Policies Beyond Explicit Reference Policies
• Fine-Tuning Masked Diffusion for Provable Self-Correction
• Finish First, Perfect Later: Test-Time Token-Level Cross-Validation for Diffusion Large Language Models
• Test-Time Scaling in Diffusion LLMs via Hidden Semi-Autoregressive Experts
• ParallelBench: Understanding the Trade-offs of Parallel Decoding in Diffusion LLMs
• AtomWorld: A Benchmark for Evaluating Spatial Reasoning in Large Language Models on Crystalline Materials
• Asynchronous Denoising Diffusion Models for Aligning Text-to-Image Generation
• LaDiR: Latent Diffusion Enhances LLMs for Text Reasoning
• Demystifying MaskGIT Sampler and Beyond: Adaptive Order Selection in Masked Diffusion
• Self Speculative Decoding for Diffusion Large Language Models
• Beyond Next-Token Prediction: A Performance Characterization of Diffusion versus Autoregressive Language Models
• Rainbow Padding: Mitigating Early Termination in Instruction-Tuned Diffusion LLMs
• OneFlow: Concurrent Mixed-Modal and Interleaved Generation with Edit Flows
• DMark: Order-Agnostic Watermarking for Diffusion Large Language Models
• Consolidating Reinforcement Learning for Multimodal Discrete Diffusion Models
• Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasoner
• In-Situ Tweedie Discrete Diffusion Models
• Syntax-Guided Diffusion Language Models with User-Integrated Personalization
• Toward Safer Diffusion Language Models: Discovery and Mitigation of Priming Vulnerability
• Training Large Language Models To Reason In Parallel With Global Forking Tokens
• Continuously Augmented Discrete Diffusion model for Categorical Generative Modeling
• Free Draft-and-Verification: Toward Lossless Parallel Decoding for Diffusion Large Language Models
• dParallel: Learnable Parallel Decoding for dLLMs
• AdaBlock-dLLM: Semantic-Aware Diffusion LLM Inference via Adaptive Block Size
• Data-to-Energy Stochastic Dynamics
• Fast-dLLM v2: Efficient Block-Diffusion LLM
• dVLA: Diffusion Vision-Language-Action Model with Multimodal Chain-of-Thought
• TraceDet: Hallucination Detection from the Decoding Trace of Diffusion Large Language Models
• DiffuGuard: How Intrinsic Safety is Lost and Found in Diffusion Large Language Models
• Watermarking Diffusion Language Models
• TR2-D2: Tree Search Guided Trajectory-Aware Fine-Tuning for Discrete Diffusion
• Ultra-Fast Language Generation via Discrete Diffusion Divergence Instruct
• Learning to Parallel: Accelerating Diffusion Large Language Models via Learnable Parallel Decoding
• Sequential Diffusion Language Models
• Taming Masked Diffusion Language Models via Consistency Trajectory Reinforcement Learning with Fewer Decoding Step
• DiffuSpec: Unlocking Diffusion Language Models for Speculative Decoding
• Don't Settle Too Early: Self-Reflective Remasking for Diffusion Language Models
• Training Optimal Large Diffusion Language Models
• RIV: Recursive Introspection Mask Diffusion Vision Language Model
• Planner Aware Path Learning in Diffusion Language Models Training
• A2D: Any-Order, Any-Step Safety Alignment for Diffusion Language Models
• Tree Reward-Aligned Search for TReASURe in Masked Diffusion Language Models
• CoDA: Coding LM via Diffusion Adaptation
• Quant-dLLM: Post-Training Extreme Low-Bit Quantization for Diffusion Large Language Models
• d²Cache: Accelerating Diffusion-Based LLMs via Dual Adaptive Caching
• Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action Spaces
• Soft-Di[M]O: Improving One-Step Discrete Image Generation with Soft Embeddings
• CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning
• Scale-Wise VAR is Secretly Discrete Diffusion
• Decoding quantum low density parity check codes with diffusion
• Generation Properties of Stochastic Interpolation under Finite Training Set
• Discrete Guidance Matching: Exact Guidance for Discrete Flow Matching
• Error Analysis of Discrete Flow with Generator Matching
• On the Complexity Theory of Masked Discrete Diffusion: From poly(1/ε) to Nearly ε-Free
• Enabling Approximate Joint Sampling in Diffusion LMs
• Deterministic Discrete Denoising
• d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimation
• FS-DFM: Fast and Accurate Long Text Generation with Few-Step Diffusion Language Models
• Discrete Diffusion for Reflective Vision-Language-Action Models in Autonomous Driving
• From Text to Talk: Audio-Language Model Needs Non-Autoregressive Joint Training
• Discrete Diffusion for Generative Modeling of Text-Aligned Speech Tokens
• PIRF: Physics-Informed Reward Fine-Tuning for Diffusion Models
• Where and When to Commit: Candidate-Aware Decoding for Diffusion Language Models
• ELF: Embedded Language Flows
• Dream-Cubed: Controllable Generative Modeling in Minecraft by Training on Billions of Cubes
• Visual Instruction Pretraining for Domain-Specific Foundation Models
• Audio-Conditioned Diffusion LLMs for ASR and Deliberation Processing
• Diffusion-Based Cross-Modal Feature Extraction for Multi-Label Classification
• Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning
• Masked Diffusion Models as Energy Minimization
• DSFT: Inspiring Diffusion Large Language Models to Comprehend Mathematical and Logical Patterns
• LLaDA-VLA: Vision Language Diffusion Action Models
• COGITAO: A Visual Reasoning Framework To Study Compositionality & Generalization
• Set Block Decoding is a Language Model Inference Accelerator
• Diffusion Generative Models Meet Compressed Sensing, with Applications to Imaging and Finance
• Dream-Coder 7B: An Open Diffusion Language Model for Code
• Any-Order Flexible Length Masked Diffusion
• Tabular Diffusion Counterfactual Explanations
• Universal Properties of Activation Sparsity in Modern Large Language Models
• Discrete Diffusion VLA: Bringing Discrete Diffusion to Action Decoding in Vision-Language-Action Policies
• Diffusion Language Models Know the Answer Before Decoding
• Blockwise SFT for Diffusion Language Models: Reconciling Bidirectional Attention and Autoregressive Decoding
• CrossHOI-Bench: A Unified Benchmark for HOI Evaluation across Vision-Language Models and HOI-Specific Methods
• Dream 7B: Diffusion Large Language Models
• Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMs
• DPad: Efficient Diffusion Language Models with Suffix Dropout
• Reinforced Context Order Recovery for Adaptive Reasoning and Planning
• ViLaD: A Large Vision Language Diffusion Framework for End-to-End Autonomous Driving
• MDPO: Overcoming the Training-Inference Divide of Masked Diffusion Language Models
• Navigating the Exploration-Exploitation Tradeoff in Inference-Time Scaling of Diffusion Models
• Improving Text Style Transfer using Masked Diffusion Language Models with Inference-time Scaling
• MDNS: Masked Diffusion Neural Sampler via Stochastic Optimal Control
• Constrained Decoding of Diffusion LLMs with Context-Free Grammars
• Time Is a Feature: Exploiting Temporal Dynamics in Diffusion Language Models
• Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing
• One Model for All: Unified Try-On and Try-Off in Any Pose via LLM-Inspired Bidirectional Tweedie Diffusion
• Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference
• LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understanding
• Beyond Fixed: Training-Free Variable-Length Denoising for Diffusion Large Language Models
• Diffusion Beats Autoregressive in Data-Constrained Settings
• Rethinking LLM Inference Bottlenecks: Insights from Latent Attention and Mixture-of-Experts
• MLLM-based Speech Recognition: When and How is Multimodality Beneficial?
• Wide-In, Narrow-Out: Revokable Decoding for Efficient and Effective DLLMs
• Continuous Diffusion Models Can Obey Formal Syntax
• Language Models for Controllable DNA Sequence Design
• The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models
• The Serial Scaling Hypothesis
• Compositional Discrete Latent Code for High Fidelity, Productive Diffusion Models
• A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Search
• Can You Detect the Difference?
• Theory-Informed Improvements to Classifier-Free Guidance for Discrete Diffusion Models
• Inference-Time Scaling of Diffusion Language Models with Particle Gibbs Sampling
• Token-Based Audio Inpainting via Discrete Diffusion
• Improving scDiffusion with Sparsity-Biased Classifier-Free Guidance
• Generative Deep Learning for de Novo Drug Design─A Chemical Space Odyssey
• A Survey on Latent Reasoning
• wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models
• Discrete Diffusion Trajectory Alignment via Stepwise Decomposition
• Unveiling the Potential of Diffusion Large Language Model in Controllable Generation
• Predict-then-Diffuse: Adaptive Response Length for Compute-Budgeted Inference in Diffusion LLMs
• Dynamic Execution Commitment of Vision-Language-Action Models
• Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMs
• Frontiers of Generative AI for Network Optimization: Theories, Limits, and Visions
• Edit Flows: Flow Matching with Edit Operations
• A "Good" Regulator May Provide a World Model for Intelligent Systems
• Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture
• Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spaces
• Plan for Speed: Dilated Scheduling for Masked Diffusion Language Models
• Diffusion Sequence Models for Enhanced Protein Representation and Generation
• LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMs
• Discrete Diffusion in Large Language and Multimodal Models: A Survey
• ContextBench: Modifying Contexts for Targeted Latent Activation
• What Exactly Does Guidance Do in Masked Discrete Diffusion Models
• Multiverse: Your Language Models Secretly Decide How to Parallelize and Merge Generation
• Escaping Confidence Trap: Evolutionary Decoding for Mathematical Reasoning in Diffusion LLMs
• Esoteric Language Models: A Family of Any-Order Diffusion LLMs
• Accelerating Diffusion LLMs via Adaptive Parallel Decoding
• DLM-One: Diffusion Language Models for One-Step Sequence Generation
• Diffusion-Based Symbolic Regression
• Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmasking
• Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Model
• DINGO: Constrained Inference for Diffusion LLMs
• Inference-Time Scaling of Discrete Diffusion Models via Importance Weighting and Optimal Proposal Design
• Score Replacement with Bounded Deviation for Rare Prompt Generation
• Breaking AR's Sampling Bottleneck: Provable Acceleration via Diffusion Language Models
• FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusion
• Discrete Markov Bridge
• ReDDiT: Rehashing Noise for Discrete Visual Generation
• Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities
• Adaptive Classifier-Free Guidance via Dynamic Low-Confidence Masking
• FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocities
• LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models
• A Framework for Stochastic Differentiable Programming
• Do Large Language Models (Really) Need Statistical Foundations?
• Anchored Diffusion Language Model
• Test-Time Scaling of Diffusion Models via Noise Trajectory Search
• Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Masking
• Alignment and Safety of Diffusion Models via Reinforcement Learning and Reward Modeling: A Survey
• Beyond On-Policy Exploration: Integrating External Policy Rollouts for Reinforcement Learning in Diffusion Language Models
• Bigger Isn't Always Memorizing: Early Stopping Overparameterized Diffusion Models
• LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning
• From Prediction to Perfection: Introducing Refinement to Autoregressive Image Generation
• MMaDA: Multimodal Large Diffusion Language Models
• dKV-Cache: The Cache for Diffusion Language Models
• Training Latent Diffusion Models with Interacting Particle Algorithms
• dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Caching
• Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Models
• TransDiffuser: Diverse Trajectory Generation with Decorrelated Multi-modal Representation for End-to-end Autonomous Driving
• Exploring More to Solve More: Boosting Diversity in Text Diffusion Models via Entropy-Based Guidance
• Plug-and-Play Guidance for Discrete Diffusion Models via Gradient-Informed Logit Correction
• HERALD: High-Throughput Block Diffusion LLM Serving via CPU-GPU Cooperative KV Cache Retrieval
• Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metrics
• DLLM-TTS: Block Discrete Diffusion Language Model for Text-to-Speech Synthesis
• Expert-Choice Routing Enables Adaptive Computation in Diffusion Language Models
• A Comparative analysis of Layer-wise Representational Capacity in AR and Diffusion LLMs
• TEAM: Temporal-Spatial Consistency Guided Expert Activation for MoE Diffusion Language Model Acceleration
• Just on Time: Token-Level Early Stopping for Diffusion Language Models
• Diffusion Language Models: An Experimental Analysis
• Speculative Correction: Draft-then-Refine Decoding for Diffusion Language Models
• Improved Large Language Diffusion Models
• Latent Reasoning with Normalizing Flows
• Diffusion LLMs can think EoS-by-EoS
• Generalized Discrete Diffusion from Snapshots
• The Design Space of Tri-Modal Masked Diffusion Models
• AMix-2: Establishing Protein as a Native Modality in Large Language Models
• LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding
• Ligand-Conditioned Discrete Diffusion for Protein Sequence-Structure Co-Design
• Categorical Flow Maps
• Reviving Any-Subset Autoregressive Models with Principled Parallel Sampling and Speculative Decoding
• Low-Rank Adaptation Redux for Large Models
• Efficient Pre-Training with Token Superposition
• LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Model
• Diffusion Language Models for Speech Recognition
• FOCUS: DLLMs Know How to Tame Their Compute Bound
• Simulation-free and finite-time diffusion model
• DDSynth-RL: Audio Synthesizer Inversion via Discrete Diffusion with Reinforcement Learning
• LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models
• Masked diffusion enables coherent beat tracking
• DELTA-TTS: Adapting Autoregressive Model into Diffusion Language Model for Text-to-Speech
• Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
• d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning
• Answer First, Reason Later: Commitment Order in Diffusion LLMs
• MMaDA-VLA: Large Diffusion Vision-Language-Action Model with Unified Multi-Modal Instruction and Generation
• d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillation
• Unifying Autoregressive and Diffusion-Based Sequence Generation

Discussions

• what the fuck shouting "ENHANCE" repeatedly at a next word predictor gets you much more rounded synthetic intellect scores arxiv.org/pdf/2502.09992 [bsky, 25 points, 7 comments]
• LLaDA - Large Language Diffusion Models arxiv.org/abs/2502.09992 Significant progress towards language diffusion models. Reportedly on par with LLaMA3 on many benchmarks. [bsky, 9 points, 2 comments]
• Wow Meta trained a version of EvoDiff for natural language! arxiv.org/abs/2502.09992 [bsky, 7 points, 0 comments]
• Large Language Diffusion Models [hn, 7 points, 3 comments]
• arxiv.org/abs/2502.09992 [bsky, 4 points, 0 comments]
• Check out the paper here, it’s an interesting way of thinking: arxiv.org/pdf/2502.09992 [bsky, 4 points, 0 comments]
• Large Language Diffusion Models [lobsters, 3 points, 0 comments]
• Interesting tid bit, uniformly sampling t then doing coinflips at prob t per step ala LLaDA arxiv.org/abs/2502.09992 is effectively balancing over all possible masks. See bottom of this post from Tim [bsky, 3 points, 1 comments]
• Large Language Diffusion Models [hn, 2 points, 0 comments]
• Diffusion LLM Has Arrived [hn, 2 points, 0 comments]
• Large Language Diffusion Models [hn, 2 points, 0 comments]
• A group of Chinese researchers have developed this model called 'LLaDA' or Large Language Diffusion model. Their 8b parameter model is on par with much larger LLM's. paper: arxiv.org/pdf/2502.09992 [bsky, 1 points, 1 comments]
• Yes ... and this visualization of the diffusion model's "thinking" is perhaps much more honest than the verbalization or inner dialog we see with exposed 01, 03 etc reasoning traces. arxiv.org/abs/25 [bsky, 1 points, 0 comments]
• Learn more here ➡️ arxiv.org/abs/2502.09992 [bsky, 0 points, 0 comments]
• If it’s Thursday, I’m at ADA (thanks Gordon!) for a solid talk. arxiv.org/abs/2502.09992 Thanks to @suhaspai.bsky.social for organizing! [bsky, 0 points, 0 comments]
• 2 links about LLdMs paper: arxiv.org/pdf/2502.09992 startup: www.inceptionlabs.ai (they are independent) [bsky, 0 points, 0 comments]
• And here is the original paper, published a few weeks ago. It comes with an open-weight 8B model, ready to test on HugginFace. buff.ly/0D00Al1 [bsky, 0 points, 0 comments]
• arxiv.org/abs/2502.09992 [bsky, 0 points, 0 comments]
• Large Language Diffusion Models https://lobste.rs/s/ibx3dt #ai [bsky, 0 points, 0 comments]

Related

• Improved Large Language Diffusion Models
• Diffusion Language Models: An Experimental Analysis
• Scaling up Masked Diffusion Models on Text
• Dream 7B: Diffusion Large Language Models
• Beyond Autoregression: Fast LLMs via Self-Distillation Through Time