Decoding Large Language Diffusion Models with Foreseeing Movement

2025/12/03 by Mo, Yichuan, Chen, Quan, Li, Mingjie +2
Computer Science · Social Sciences · #Computational and Text Analysis Methods #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Topic Modeling

paper · doi:10.48550/arxiv.2512.04135

openalex publication_date 2025/12/03 · openalex created_date 2025/12/06 · openalex updated_date 2026/07/28

Abstract

Large Language Diffusion Models (LLDMs) benefit from a flexible decoding mechanism that enables parallelized inference and controllable generations over autoregressive models. Yet such flexibility introduces a critical challenge: inference performance becomes highly sensitive to the decoding order of tokens. Existing heuristic methods, however, focus mainly on local effects while overlooking long-term impacts. To address this limitation, we propose the Foreseeing Decoding Method (FDM), a novel approach that integrates both local and global considerations to unlock the full potential, employing a search-based strategy to enable effective optimization in discrete spaces. Furthermore, by analyzing the consistency of chosen tokens in the full decoding process, we develop a variant, FDM with Acceleration (FDM-A), which restricts deep exploration to critical steps identified as the exploration and balance circumantences. Extensive experiments across diverse benchmarks and model architectures validate the scalability of FDM and demonstrate the superior efficiency-performance trade-off achieved by FDM-A. Our work might potentially provide a principled step toward more powerful decoding methods for LLDMs.

Citations

• Dream-Coder 7B: An Open Diffusion Language Model for Code
• Dream 7B: Diffusion Large Language Models
• A Survey on Diffusion Language Models
• LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understanding
• Kimi K2: Open Agentic Intelligence
• Wide-In, Narrow-Out: Revokable Decoding for Efficient and Effective DLLMs
• Unveiling the Potential of Diffusion Large Language Model in Controllable Generation
• Mercury: Ultra-Fast Language Models Based on Diffusion
• Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmasking
• Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding
• Breaking AR's Sampling Bottleneck: Provable Acceleration via Diffusion Language Models
• FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusion
• LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models
• LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning
• dKV-Cache: The Cache for Diffusion Language Models
• MMaDA: Multimodal Large Diffusion Language Models
• d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning
• Large Language Diffusion Models
• Theoretical Benefit and Limitation of Diffusion Language Model
• Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions
• Path Planning for Masked Diffusion Model Sampling
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Forest-of-Thought: Scaling Test-Time Compute for Enhancing LLM Reasoning
• Scaling Diffusion Language Models via Adaptation from Autoregressive Models
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
• The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"
• A Reparameterized Discrete Diffusion Model for Text Generation
• Scalable Diffusion Models with Transformers
• DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models
• Diffusion-LM Improves Controllable Text Generation
• High-Resolution Image Synthesis with Latent Diffusion Models
• Training Verifiers to Solve Math Word Problems
• Evaluating Large Language Models Trained on Code
• Structured Denoising Diffusion Models in Discrete State-Spaces
• Denoising Diffusion Probabilistic Models
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• OpenAI o1 System Card

Related

• From Bits to Rounds: Parallel Decoding with Exploration for Diffusion Language Models
• Accelerating Diffusion LLMs via Adaptive Parallel Decoding
• Self Speculative Decoding for Diffusion Large Language Models
• Orchestrating Dual-Boundaries: An Arithmetic Intensity Inspired Acceleration Framework for Diffusion Language Models
• Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning