Fine-Tuning Discrete Diffusion Models with Policy Gradient Methods

2025/02/03 by Oussama Zekri, Oussama Zékri, Nicolas Boullé +2 · 1 voice · 11 citations
Economics, Econometrics and Finance · #Climate Change Policy and Economics #cs.AI #cs.CL #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2502.01384

openalex publication_date 2025/02/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/29

Abstract

Discrete diffusion models have recently gained significant attention due to their ability to process complex discrete structures for language modeling. However, fine-tuning these models with policy gradient methods, as is commonly done in Reinforcement Learning from Human Feedback (RLHF), remains a challenging task. We propose an efficient, broadly applicable, and theoretically justified policy gradient algorithm, called Score Entropy Policy Optimization (\SEPO), for fine-tuning discrete diffusion models over non-differentiable rewards. Our numerical experiments across several discrete generative tasks demonstrate the scalability and efficiency of our method. Our code is available at https://github.com/ozekri/SEPO.

Cited by

• Temporal Difference Learning for Diffusion Models
• Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspective
• Understanding Sampler Stochasticity in Training Diffusion Models for RLHF
• Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimization
• Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimization
• From Data to Rewards: a Bilevel Optimization Perspective on Maximum Likelihood Estimation
• DiFFPO: Training Diffusion LLMs to Reason Fast and Furious via Reinforcement Learning
• Improving Discrete Diffusion Unmasking Policies Beyond Explicit Reference Policies
• TR2-D2: Tree Search Guided Trajectory-Aware Fine-Tuning for Discrete Diffusion
• Ultra-Fast Language Generation via Discrete Diffusion Divergence Instruct
• Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action Spaces

Discussions

• 🚀 Policy gradient methods like DeepSeek’s GRPO are great for finetuning LLMs via RLHF. But what happens when we swap autoregressive generation for discrete diffusion, a rising architecture promising [bsky, 6 points, 1 comments]

Related

• A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Models
• Directly Fine-Tuning Diffusion Models on Differentiable Rewards
• Flow Matching Policy Gradients
• Consolidating Reinforcement Learning for Multimodal Discrete Diffusion Models
• Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning