Diffusion Actor-Critic with Entropy Regulator

2024/05/24 by Yinuo Wang, Wang, Yinuo, Likun Wang +19 · 16 citations
Physics and Astronomy · #Advanced Thermodynamics and Statistical Mechanics #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2405.15177

openalex publication_date 2024/05/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Reinforcement learning (RL) has proven highly effective in addressing complex decision-making and control tasks. However, in most traditional RL algorithms, the policy is typically parameterized as a diagonal Gaussian distribution with learned mean and variance, which constrains their capability to acquire complex policies. In response to this problem, we propose an online RL algorithm termed diffusion actor-critic with entropy regulator (DACER). This algorithm conceptualizes the reverse process of the diffusion model as a novel policy function and leverages the capability of the diffusion model to fit multimodal distributions, thereby enhancing the representational capacity of the policy. Since the distribution of the diffusion policy lacks an analytical expression, its entropy cannot be determined analytically. To mitigate this, we propose a method to estimate the entropy of the diffusion policy utilizing Gaussian mixture model. Building on the estimated entropy, we can learn a parameter α that modulates the degree of exploration and exploitation. Parameter α will be employed to adaptively regulate the variance of the added noise, which is applied to the action output by the diffusion model. Experimental trials on MuJoCo benchmarks and a multimodal task demonstrate that the DACER algorithm achieves state-of-the-art (SOTA) performance in most MuJoCo control tasks while exhibiting a stronger representational capacity of the diffusion policy.

Cited by

• One-Step Generative Policies with Q-Learning: A Reformulation of MeanFlow
• Learning Intractable Multimodal Policies with Reparameterization and Diversity Regularization
• Controllable Flow Matching for Online Reinforcement Learning
• Off-policy Reinforcement Learning with Model-based Exploration Augmentation
• Mind Your Entropy: From Maximum Entropy to Trajectory Entropy-Constrained RL
• DeepAgent: A General Reasoning Agent with Scalable Toolsets
• Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Control
• A Diffusion-Refined Planner with Reinforcement Learning Priors for Confined-Space Parking
• Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action Spaces
• Actor-Critic without Actor
• D3P: Dynamic Denoising Diffusion Policy via Reinforcement Learning
• One-Step Flow Policy Mirror Descent
• From Seeing to Experiencing: Scaling Navigation Foundation Models with Reinforcement Learning
• Consistency Trajectory Planning: High-Quality and Efficient Trajectory Optimization for Offline Model-Based Reinforcement Learning
• Distributional Soft Actor-Critic with Diffusion Policy
• Flow-Based Policy for Online Reinforcement Learning

Related

• Maximum Entropy Reinforcement Learning with Diffusion Policy
• Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learning
• Enhanced DACER Algorithm with High Diffusion Efficiency
• DIME:Diffusion-Based Maximum Entropy Reinforcement Learning
• Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning