2025/10/05 by Anthony Zhan, Zhan, Anthony · 2 citations
Computer Science · #Autoregressive model #Base (topology) #Heuristic #Inference #Language model #Limiting #Multi-Agent Systems and Negotiation #Natural Language Processing Techniques #Reinforcement learning #Semantic Web and Ontologies #Simple (philosophy) #cs.AI #cs.CL #cs.LG
paper · pdf · doi:10.48550/arxiv.2510.04019
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2025/10/05 · openalex created_date 2025/10/09 · openalex updated_date 2026/08/01
Diffusion large language models (dLLMs) represent a promising alternative to autoregressive LLMs; however, the lack of effective post-training techniques, including reinforcement learning (RL), remains a key challenge for dLLMs, especially for downstream applications. Existing approaches often rely on a sequence-level view that requires biased likelihood approximations. In this work, we propose Amortized Group Relative Policy Optimization (AGRPO), a policy gradient algorithm that leverages the Markovian nature of dLLMs, optimizing individual denoising steps rather than full sequences. Our approach improves alignment between the trained policy and the inference process and also admits efficient, unbiased gradient updates via a novel timestep estimation scheme. We demonstrate AGRPO's effectiveness on different math and reasoning tasks, achieving absolute accuracy gains of +59.4% and +69.7% on Countdown and Sudoku over the base LLaDA model, exceeding comparable methods such as diffu-GRPO. Code is available at https://github.com/probablyabot/agrpo.