Reinforcement Learning by Value Gradients

2008/03/25 by Michael Fairbank, Fairbank, Michael
Computer Science · #Adaptive Dynamic Programming Control #Artificial Intelligence (cs.AI) #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Neural and Evolutionary Computing (cs.NE) #Reinforcement Learning in Robotics #cs.AI #cs.NE

paper · pdf · doi:10.48550/arxiv.0803.3539

arxiv created 2008/03/25 · openalex publication_date 2008/03/25 · arxiv updated 2009/12/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The concept of the value-gradient is introduced and developed in the context of reinforcement learning. It is shown that by learning the value-gradients exploration or stochastic behaviour is no longer needed to find locally optimal trajectories. This is the main motivation for using value-gradients, and it is argued that learning value-gradients is the actual objective of any value-function learning algorithm for control problems. It is also argued that learning value-gradients is significantly more efficient than learning just the values, and this argument is supported in experiments by efficiency gains of several orders of magnitude, in several problem domains. Once value-gradients are introduced into learning, several analyses become possible. For example, a surprising equivalence between a value-gradient learning algorithm and a policy-gradient learning algorithm is proven, and this provides a robust convergence proof for control problems using a value function with a general function approximator.

Related

• The Local Optimality of Reinforcement Learning by Value Gradients, and its Relationship to Policy Gradient Learning
• A General Control-Theoretic Approach for Reinforcement Learning: Theory and Algorithms
• Policy Gradient using Weak Derivatives for Reinforcement Learning
• Policy Gradient for Reinforcement Learning with General Utilities
• Orchestrated Value Mapping for Reinforcement Learning