Learning Continuous Control Policies by Stochastic Value Gradients

2015/10/30 by Nicolas Heess, Greg Wayne, Heess, Nicolas +9 · 112 citations
Computer Science · Engineering · Mathematics · #Advanced Control Systems Optimization #Artificial intelligence #Computer science #Control (management) #Economics #FOS: Computer and information sciences #Fault Detection and Control Systems #Machine Learning (cs.LG) #Mathematical economics #Mathematical optimization #Mathematics #Neural and Evolutionary Computing (cs.NE) #Optimal control #Reinforcement Learning in Robotics #Statistics #Stochastic control #Value (mathematics) #cs.LG #cs.NE

paper · pdf · doi:10.48550/arxiv.1510.09142

published in arXiv (Cornell University) (Cornell University) · 13 pages, NIPS 2015

arxiv created 2015/10/30 · openalex publication_date 2015/10/30 · arxiv updated 2015/11/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present a unified framework for learning continuous control policies using backpropagation. It supports stochastic control by treating stochasticity in the Bellman equation as a deterministic function of exogenous noise. The product is a spectrum of general policy gradient algorithms that range from model-free methods with value functions to model-based methods without value functions. We use learned models but only require observations from the environment in- stead of observations from model-predicted trajectories, minimizing the impact of compounded model errors. We apply these algorithms first to a toy stochastic control problem and then to several physics-based control problems in simulation. One of these variants, SVG(1), shows the effectiveness of learning models, value functions, and policies simultaneously in continuous domains.

Citations

• On the difficulty of training Recurrent Neural Networks
• Identification and control of dynamical systems using neural networks
• Compatible Value Gradients for Reinforcement Learning of Continuous Deep Policies

Cited by

• SDE Guided Monte Carlo Reinforcement Learning: A Stochastic Maximum Principle Approach for Robust Decision Making in Noisy Environments
• Spectral Representation-based Reinforcement Learning
• Double Horizon Model-Based Policy Optimization
• Prismatic World Model: Learning Compositional Dynamics for Planning in Hybrid Systems
• D2 Actor Critic: Diffusion Actor Meets Distributional Critic
• Relative Entropy Regularized Policy Iteration
• Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
• Graph networks as learnable physics engines for inference and control
• Memory-Augmented Potential Field Theory: A Framework for Adaptive Control in Non-Convex Domains
• Hybrid Actor-Critic Reinforcement Learning in Parameterized Action Space
• First Order Model-Based RL through Decoupled Backpropagation
• Beyond Prediction: Reinforcement Learning as the Defining Leap in Healthcare AI
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Multitask Soft Option Learning
• Reparameterization Proximal Policy Optimization
• Model-Based Regularization for Deep Reinforcement Learning with Transcoder Networks
• High-Dimensional Continuous Control Using Generalized Advantage\n Estimation
• Test-time Offline Reinforcement Learning on Goal-related Experience
• Neuro-Inspired Inverse Learning for Planning and Control
• Sample Efficient Actor-Critic with Experience Replay
• AirLLM: Diffusion Policy-based Adaptive LoRA for Remote Fine-Tuning of LLM over the Air
• Multi-residual Mixture of Experts Learning for Cooperative Control in Multi-vehicle Systems
• Recall Traces: Backtracking Models for Efficient Reinforcement Learning
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Distributions as Actions: A Unified Framework for Diverse Action Spaces
• Fourier Policy Gradients
• MBMF: Model-Based Priors for Model-Free Reinforcement Learning
• DyNODE: Neural Ordinary Differential Equations for Dynamics Modeling in Continuous Control
• AMOR: Adaptive Character Control through Multi-Objective Reinforcement Learning
• Learning to Reweight Imaginary Transitions for Model-Based Reinforcement Learning
• Smoothed Action Value Functions for Learning Gaussian Policies
• Wasserstein Policy Optimization
• On the Role of Computation in Reinforcement Learning
• Robust Control under Stationary Ambiguity
• The wireless control plane: An overview and directions for future research

Related

• Learning Optimal Deterministic Policies with Stochastic Policy Gradients
• Mean-Field Generalisation Bounds for Learning Controls in Stochastic Environments
• Operator Models for Continuous-Time Offline Reinforcement Learning
• Deterministic Value-Policy Gradients
• Continuous Policy and Value Iteration for Stochastic Control Problems and Its Convergence