Deterministic Discrete Denoising

2025/09/25 by Hideyuki Suzuki, Suzuki, Hideyuki, Kurebayashi, Wataru +1
Engineering · #Advanced Numerical Analysis Techniques #Chaotic Dynamics (nlin.CD) #FOS: Computer and information sciences #FOS: Physical sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2509.20896

openalex publication_date 2025/09/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose a deterministic denoising algorithm for discrete-state diffusion models. The key idea is to derandomize the generative reverse Markov chain by introducing a variant of the herding algorithm, which induces deterministic state transitions driven by weakly chaotic dynamics. It serves as a direct replacement for the stochastic denoising process, without requiring retraining or continuous state embeddings. We demonstrate consistent improvements in both efficiency and sample quality on text and image generation tasks. In addition, the proposed algorithm yields improved solutions for diffusion-based combinatorial optimization. Thus, herding-based denoising is a simple yet promising approach for enhancing the generative process of discrete diffusion models. Furthermore, our results reveal that deterministic reverse processes, well established in continuous diffusion, can also be effective in discrete state spaces.

Citations

• Scaling Behavior of Discrete Diffusion Language Models
• Discrete Diffusion Models: Novel Analysis and New Sampler Guarantees
• The Diffusion Duality
• Absorb and Converge: Provable Convergence Guarantee for Absorbing Discrete Diffusion Models
• Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmasking
• LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models
• Generalized Interpolating Discrete Diffusion
• Large Language Diffusion Models
• Debiasing Guidance for Discrete Diffusion with Sequential Monte Carlo
• Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithms
• A General Framework for Inference-time Scaling and Steering of Diffusion Models
• Simple Guidance Mechanisms for Discrete Diffusion Models
• Jump Your Steps: Optimizing Sampling Schedule of Discrete Diffusion Models
• How Discrete and Continuous Diffusion Meet: Comprehensive Analysis of Discrete Diffusion Models via a Stochastic Integral Framework
• Simple and Effective Masked Diffusion Language Models
• Simplified and Generalized Masked Diffusion for Discrete Data
• Fast Sampling via Discrete Non-Markov Diffusion Models with Predetermined Transition Time
• Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution
• DIFUSCO: Graph-based Diffusion Solvers for Combinatorial Optimization
• Score-based Continuous-time Discrete Diffusion Models
• Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning
• A Continuous Time Framework for Discrete Denoising Models
• Structured Denoising Diffusion Models in Discrete State-Spaces
• Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions
• MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers
• Denoising Diffusion Implicit Models
• Denoising Diffusion Probabilistic Models
• Improved Techniques for Training GANs
• Rethinking the Inception Architecture for Computer Vision
• U-Net: Convolutional Networks for Biomedical Image Segmentation
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling
• Optimally-Weighted Herding is Bayesian Quadrature

Related

• Structured Denoising Diffusion Models in Discrete State-Spaces
• The Missing U for Efficient Diffusion Models
• Blackout Diffusion: Generative Diffusion Models in Discrete-State Spaces
• Continuously Augmented Discrete Diffusion model for Categorical Generative Modeling
• Denoising Diffusion Implicit Models