Towards Learning Controllable Representations of Physical Systems

2020/11/16 by Kevin Haninger, Haninger, Kevin, Raul Vicente-Garcia +2
Computer Science · Engineering · Physics and Astronomy · #Advanced Control Systems Optimization #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Model Reduction and Neural Networks #Reinforcement Learning in Robotics #Robotics (cs.RO) #Systems and Control (eess.SY) #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2011.09906

openalex publication_date 2020/11/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Learned representations of dynamical systems reduce dimensionality, potentially supporting downstream reinforcement learning (RL). However, no established methods predict a representation's suitability for control and evaluation is largely done via downstream RL performance, slowing representation design. Towards a principled evaluation of representations for control, we consider the relationship between the true state and the corresponding representations, proposing that ideally each representation corresponds to a unique true state. This motivates two metrics: temporal smoothness and high mutual information between true state/representation. These metrics are related to established representation objectives, and studied on Lagrangian systems where true state, information requirements, and statistical properties of the state can be formalized for a broad class of systems. These metrics are shown to predict reinforcement learning performance in a simulated peg-in-hole task when comparing variants of autoencoder-based representations.

Citations

• Slow Feature Analysis: Unsupervised Learning of Invariances
• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Learning Latent Dynamics for Planning from Pixels
• Reinforcement Learning on Variable Impedance Controller for High-Precision Robotic Assembly

Related

• Control-Aware Representations for Model-based Reinforcement Learning
• Dynamics-aware Embeddings
• S-RL Toolbox: Environments, Datasets and Evaluation Metrics for State Representation Learning
• On the Generalization of Representations in Reinforcement Learning
• Unsupervised Representation Learning in Deep Reinforcement Learning: A Review