Learning Continuous Control Policies by Stochastic Value Gradients

2015/10/30 by Nicolas Heess, Greg Wayne, Heess, Nicolas +9 · 17 citations
Computer Science · Engineering · #Advanced Control Systems Optimization #FOS: Computer and information sciences #Fault Detection and Control Systems #Machine Learning (cs.LG) #Neural and Evolutionary Computing (cs.NE) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.1510.09142

openalex publication_date 2015/10/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present a unified framework for learning continuous control policies using backpropagation. It supports stochastic control by treating stochasticity in the Bellman equation as a deterministic function of exogenous noise. The product is a spectrum of general policy gradient algorithms that range from model-free methods with value functions to model-based methods without value functions. We use learned models but only require observations from the environment in- stead of observations from model-predicted trajectories, minimizing the impact of compounded model errors. We apply these algorithms first to a toy stochastic control problem and then to several physics-based control problems in simulation. One of these variants, SVG(1), shows the effectiveness of learning models, value functions, and policies simultaneously in continuous domains.

Citations

• On the difficulty of training Recurrent Neural Networks
• Identification and control of dynamical systems using neural networks
• Compatible Value Gradients for Reinforcement Learning of Continuous Deep\n Policies

Cited by

• SDE Guided Monte Carlo Reinforcement Learning: A Stochastic Maximum Principle Approach for Robust Decision Making in Noisy Environments
• Spectral Representation-based Reinforcement Learning
• Double Horizon Model-Based Policy Optimization
• Prismatic World Model: Learning Compositional Dynamics for Planning in Hybrid Systems
• D2 Actor Critic: Diffusion Actor Meets Distributional Critic
• Relative Entropy Regularized Policy Iteration
• Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
• Graph networks as learnable physics engines for inference and control
• Memory-Augmented Potential Field Theory: A Framework for Adaptive Control in Non-Convex Domains
• Hybrid Actor-Critic Reinforcement Learning in Parameterized Action Space
• First Order Model-Based RL through Decoupled Backpropagation
• Beyond Prediction: Reinforcement Learning as the Defining Leap in Healthcare AI
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Multitask Soft Option Learning
• Reparameterization Proximal Policy Optimization
• Model-Based Regularization for Deep Reinforcement Learning with Transcoder Networks
• Test-time Offline Reinforcement Learning on Goal-related Experience

Related

• Learning Optimal Deterministic Policies with Stochastic Policy Gradients
• Mean-Field Generalisation Bounds for Learning Controls in Stochastic Environments
• Operator Models for Continuous-Time Offline Reinforcement Learning
• Deterministic Value-Policy Gradients
• Continuous Policy and Value Iteration for Stochastic Control Problems and Its Convergence