Estimating Q(s,s') with Deep Deterministic Dynamics Gradients

2020/02/21 by Ashley D. Edwards, Edwards, Ashley D., Himanshu Sahni +17 · 1 citation
Computer Science · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2002.09505

openalex publication_date 2020/02/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we introduce a novel form of value function, Q(s, s'), that expresses the utility of transitioning from a state s to a neighboring state s' and then acting optimally thereafter. In order to derive an optimal policy, we develop a forward dynamics model that learns to make next-state predictions that maximize this value. This formulation decouples actions from values while still learning off-policy. We highlight the benefits of this approach in terms of value function transfer, learning within redundant action spaces, and learning off-policy from state observations generated by sub-optimal or completely random policies. Code and videos are available at http://sites.google.com/view/qss-paper.

Citations

• Hindsight Experience Replay
• Addressing Function Approximation Error in Actor-Critic Methods
• Time-Contrastive Networks: Self-Supervised Learning from Video

Cited by

• Expressive Value Learning for Scalable Offline Reinforcement Learning

Related

• Learning Value Functions from Undirected State-only Experience
• Minimum information divergence of Q-functions for dynamic treatment resumes
• Approximating two value functions instead of one: towards characterizing\n a new family of Deep Reinforcement Learning algorithms
• Accelerating Policy Gradient by Estimating Value Function from Prior Computation in Deep Reinforcement Learning
• Learning Successor States and Goal-Dependent Values: A Mathematical Viewpoint