Symmetry reduction for deep reinforcement learning active control of chaotic spatiotemporal dynamics

2021/04/09 by Kevin Zeng, Michael D. Graham
Computer Science · Engineering · Mathematics · Physics and Astronomy · #Artificial intelligence #Chaotic #Computer science #Control (management) #Control theory (sociology) #Deep learning #Fluid Dynamics and Turbulent Flows #Mathematics #Model Reduction and Neural Networks #Reduction (mathematics) #Reinforcement Learning in Robotics #Reinforcement learning #State space #Symmetry (geometry) #cs.LG #nlin.CD

paper · pdf · doi:10.1103/physreve.104.014210

published as Phys. Rev. E 104, 014210 (2021) · Submitted to Physical Review E

arxiv created 2021/04/09 · openalex publication_date 2021/07/16 · arxiv updated 2021/07/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

Deep reinforcement learning (RL) is a data-driven, model-free method capable of discovering complex control strategies for macroscopic objectives in high-dimensional systems, making its application toward flow control promising. Many systems of flow control interest possess symmetries that, when neglected, can significantly inhibit the learning and performance of a naive deep RL approach. Using a test-bed consisting of the Kuramoto-Sivashinsky equation (KSE), equally spaced actuators, and a goal of minimizing dissipation and power cost, we demonstrate that by moving the deep RL problem to a symmetry-reduced space, we can alleviate limitations inherent in the naive application of deep RL. We demonstrate that symmetry-reduced deep RL yields improved data efficiency as well as improved control policy efficacy compared to policies found by naive deep RL. Interestingly, the policy learned by the symmetry aware control agent drives the system toward an equilibrium state of the forced KSE that is connected by continuation to an equilibrium of the unforced KSE, despite having been given no explicit information regarding its existence. That is, to achieve its goal, the RL algorithm discovers and stabilizes an equilibrium state of the system. Finally, we demonstrate that the symmetry-reduced control policy is robust to observation and actuation signal noise, as well as to system parameters it has not observed before.

Citations

• Rotation-invariant convolutional neural networks for galaxy morphology prediction
• Approximation by superpositions of a sigmoidal function
• Human-level control through deep reinforcement learning
• ImageNet classification with deep convolutional neural networks
• Mastering the game of Go with deep neural networks and tree search
• Exploiting Cyclic Symmetry in Convolutional Neural Networks
• Equivariance Through Parameter-Sharing
• Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control
• Universal approximations of permutation invariant/equivariant functions by deep neural networks
• Universal Invariant and Equivariant Graph Neural Networks
• Dota 2 with Large Scale Deep Reinforcement Learning
• Data-Driven Stabilization of Periodic Orbits