Discovering Behavioral Modes in Deep Reinforcement Learning Policies Using Trajectory Clustering in Latent Space

2024/02/20 by Sindre Benjamin Remman, Remman, Sindre Benjamin, Anastasios M. Lekkas +1
Computer Science · #Anomaly Detection Techniques and Applications #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2402.12939

openalex publication_date 2024/02/20 · openalex created_date 2024/02/22 · openalex updated_date 2026/07/28

Abstract

Understanding the behavior of deep reinforcement learning (DRL) agents is crucial for improving their performance and reliability. However, the complexity of their policies often makes them challenging to understand. In this paper, we introduce a new approach for investigating the behavior modes of DRL policies, which involves utilizing dimensionality reduction and trajectory clustering in the latent space of neural networks. Specifically, we use Pairwise Controlled Manifold Approximation Projection (PaCMAP) for dimensionality reduction and TRACLUS for trajectory clustering to analyze the latent space of a DRL policy trained on the Mountain Car control task. Our methodology helps identify diverse behavior patterns and suboptimal choices by the policy, thus allowing for targeted improvements. We demonstrate how our approach, combined with domain knowledge, can enhance a policy's performance in specific regions of the state space.

Related

• From Parameters to Behavior: Unsupervised Compression of the Policy Space
• Deep Reinforcement Learning Behavioral Mode Switching Using Optimal Control Based on a Latent Space Objective
• Policy-Based Trajectory Clustering in Offline Reinforcement Learning
• Deep Reinforcement Learning Discovers Internal Models
• Composing Meta-Policies for Autonomous Driving Using Hierarchical Deep Reinforcement Learning