Dynamic Entropy Tuning in Reinforcement Learning Low-Level Quadcopter Control: Stochasticity vs Determinism

2025/12/20 by Youssef Mahran, Mahran, Youssef, Zeyad Gamal +3
Computer Science · Engineering · #Adaptive Dynamic Programming Control #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #Robotics (cs.RO) #UAV Applications and Optimization

paper · doi:10.48550/arxiv.2512.18336

openalex publication_date 2025/12/20 · openalex created_date 2025/12/24 · openalex updated_date 2026/07/28

Abstract

This paper explores the impact of dynamic entropy tuning in Reinforcement Learning (RL) algorithms that train a stochastic policy. Its performance is compared against algorithms that train a deterministic one. Stochastic policies optimize a probability distribution over actions to maximize rewards, while deterministic policies select a single deterministic action per state. The effect of training a stochastic policy with both static entropy and dynamic entropy and then executing deterministic actions to control the quadcopter is explored. It is then compared against training a deterministic policy and executing deterministic actions. For the purpose of this research, the Soft Actor-Critic (SAC) algorithm was chosen for the stochastic algorithm while the Twin Delayed Deep Deterministic Policy Gradient (TD3) was chosen for the deterministic algorithm. The training and simulation results show the positive effect the dynamic entropy tuning has on controlling the quadcopter by preventing catastrophic forgetting and improving exploration efficiency.

Citations

• A Model-free Deep Reinforcement Learning Approach To Maneuver A Quadrotor Despite Single Rotor Failure
• Learning to Fly -- a Gym Environment with PyBullet Physics for\n Reinforcement Learning of Multi-agent Quadcopter Control
• Using Soft Actor-Critic for Low-Level UAV Control
• Adaptive Online Planning for Continual Lifelong Learning
• Soft Actor-Critic Algorithms and Applications
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

Related

• Trajectory Entropy Reinforcement Learning for Predictable and Robust Control
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Mind Your Entropy: From Maximum Entropy to Trajectory Entropy-Constrained RL
• Robust Deterministic Policy Gradient for Disturbance Attenuation and Its Application to Quadrotor Control
• Examining Policy Entropy of Reinforcement Learning Agents for Personalization Tasks