Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flow

2024/05/22 by Chen-Hao Chao, Chao, Chen-Hao, Chien Feng +9 · 6 citations
Computer Science · #Anomaly Detection Techniques and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2405.13629

openalex publication_date 2024/05/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Existing Maximum-Entropy (MaxEnt) Reinforcement Learning (RL) methods for continuous action spaces are typically formulated based on actor-critic frameworks and optimized through alternating steps of policy evaluation and policy improvement. In the policy evaluation steps, the critic is updated to capture the soft Q-function. In the policy improvement steps, the actor is adjusted in accordance with the updated soft Q-function. In this paper, we introduce a new MaxEnt RL framework modeled using Energy-Based Normalizing Flows (EBFlow). This framework integrates the policy evaluation steps and the policy improvement steps, resulting in a single objective training process. Our method enables the calculation of the soft value function used in the policy evaluation target without Monte Carlo approximation. Moreover, this design supports the modeling of multi-modal action distributions while facilitating efficient action sampling. To evaluate the performance of our method, we conducted experiments on the MuJoCo benchmark suite and a number of high-dimensional robotic tasks simulated by Omniverse Isaac Gym. The evaluation results demonstrate that our method achieves superior performance compared to widely-adopted representative baselines.

Cited by

• A Theoretical Lens for RL-Tuned Language Models via Energy-Based Models
• Efficient Reinforcement Learning with Semantic and Token Entropy for LLM Reasoning
• CDFlow: Building Invertible Layers with Circulant and Diagonal Matrices
• Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Markov Likelihood
• Entropy Regularizing Activation: Boosting Continuous Control, Large Language Models, and Image Classification with Activation as Entropy Constraints
• FlowRL: Matching Reward Distributions for LLM Reasoning

Related

• Soft Policy Gradient Method for Maximum Entropy Deep Reinforcement Learning
• Reinforcement Learning with Deep Energy-Based Policies
• Maximum Entropy Reinforcement Learning with Diffusion Policy
• S²AC: Energy-Based Reinforcement Learning with Stein Soft Actor Critic
• Mind Your Entropy: From Maximum Entropy to Trajectory Entropy-Constrained RL