Data-Efficient Learning of Feedback Policies from Image Pixels using Deep Dynamical Models

2015/10/08 by John-Alexander M. Assael, Niklas Wahlström, Assael, John-Alexander M. +5 · 2 citations
Computer Science · Physics and Astronomy · #Adaptive Dynamic Programming Control #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Model Reduction and Neural Networks #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.1510.02173

openalex publication_date 2015/10/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Data-efficient reinforcement learning (RL) in continuous state-action spaces using very high-dimensional observations remains a key challenge in developing fully autonomous systems. We consider a particularly important instance of this challenge, the pixels-to-torques problem, where an RL agent learns a closed-loop control policy ("torques") from pixel information only. We introduce a data-efficient, model-based reinforcement learning algorithm that learns such a closed-loop policy directly from pixel information. The key ingredient is a deep dynamical model for learning a low-dimensional feature embedding of images jointly with a predictive model in this low-dimensional feature space. Joint learning is crucial for long-term predictions, which lie at the core of the adaptive nonlinear model predictive control strategy that we use for closed-loop control. Compared to state-of-the-art RL methods for continuous states and actions, our approach learns quickly, scales to high-dimensional state spaces, is lightweight and an important step toward fully autonomous end-to-end learning from pixels to torques.

Citations

• Adam: A Method for Stochastic Optimization
• Stochastic Backpropagation and Approximate Inference in Deep Generative\n Models
• Deep Learning in Neural Networks: An Overview
• Gradient-based learning applied to document recognition
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• End-to-End Training of Deep Visuomotor Policies
• ImageNet classification with deep convolutional neural networks
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Nonlinear Model Order Reduction of Dynamical Systems in Process Engineering: Review and Comparison
• Multi-Objective Deep Reinforcement Learning

Related

• From Pixels to Torques: Policy Learning with Deep Dynamical Models
• SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
• Self-supervised Learning of Image Embedding for Continuous Control
• Dynamics-aware Embeddings
• Low Dimensional State Representation Learning with Robotics Priors in Continuous Action Spaces